Methods and systems for managing facility power and cooling

ABSTRACT

Systems and methods are provided for determining data center cooling and power requirements and for monitoring performance of cooling and power systems in data centers. At least one aspect provides a system and method that enables a data center operator to determine available power and cooling at specific areas and enclosures in a data center to assist in locating new equipment in the data center.

RELATED APPLICATIONS

The present application is a continuation of U.S. patent applicationSer. No. 11/342,285 filed on Jan. 27, 2006, entitled METHODS AND SYSTEMSFOR MANAGING FACILITY POWER AND COOLING, which is a continuation-in-partof U.S. patent application Ser. No. 11/120,137 filed May 2, 2005,entitled METHODS AND SYSTEMS FOR MANAGING FACILITY POWER AND COOLING.U.S. patent application Ser. No. 11/342,285 filed on Jan. 27, 2006,claims priority under 35 U.S.C. §119(e) to U.S. Provisional ApplicationSer. No. 60/719,356, filed Sep. 22, 2005, entitled METHODS AND SYSTEMSFOR MANAGING FACILITY POWER AND COOLING, all of the above priorapplications are hereby incorporated herein by reference. Further, thepresent application relates to U.S. patent application Ser. No.11/342,300, entitled METHODS AND SYSTEMS FOR MANAGING FACILITY POWER ANDCOOLING, by Rasmussen et al. filed on Jan. 27, 2006, which is alsoincorporated herein by reference.

BACKGROUND OF INVENTION

1. Field of Invention

Embodiments of the invention relate generally to methods and systems formanaging facility power and cooling.

2. Discussion of Related Art

Centralized data centers for computer, communications and otherelectronic equipment have been in use for a number of years, and morerecently with the increasing use of the Internet, large scale datacenters that provide hosting services for Internet Service Providers(ISPs), Application Service Providers (ASPs) and Internet contentproviders are becoming more prevalent. Typical centralized data centerscontain numerous racks of equipment that require power, cooling andconnections to external communications facilities. In modern datacenters and network rooms, the increased density of computing equipmentused in these facilities has put strains on the cooling and powersystems of the facilities. In the past, typical power consumption foreach equipment enclosure in a data facility was on the order of 1 kW.With the use of server blades and other high power density equipment inequipment racks, it is not uncommon for an equipment rack to have apower draw of 10 kW or even as high as 25 kW.

Typically, the power consumed by computer equipment is converted to heatand typically, the cooling requirements of a facility are determinedbased on the power requirements of the facility. Typical data centersutilize air plenums under raised floors to distribute cooling airthrough a data center. One or more computer room air conditioners(CRACs) or computer room air handlers (CRAHs) are typically distributedalong the periphery of the data room, and these units draw return airfrom the room or a ceiling plenum and distribute cooling air beneath theraised floor. Perforated tiles may be placed in front or beneath racksof equipment to be cooled to allow the cooling air from beneath thefloor to cool equipment within the racks.

Several tools are available to assist a data center designer inconfiguring a layout of a data center to provide necessary power andcooling to equipment to be located in the data center. These toolstypically assist a designer in determining total power requirements andaccordingly overall cooling requirements for a data center. In addition,these tools may assist a designer in determining optimum equipmentlayout and proper sizing of power cabling and circuit breakers.

While existing tools provide a designer with detailed layout informationregarding power distribution, these tools typically provide far lesshelp in determining cooling needs for a facility. Advanced programs thatuse computational fluid dynamics (CFD) may be used to model the coolingdesign of a facility, but the use of such programs is extremely limiteddue to the complexity of the programs, which results in their use beingprohibitively expensive and time consuming. U.S. Patent ApplicationUS2003/0158718 A1 to Nakagawa et al describes an automated system fordesigning a cooling system for a facility. In the system of Nakagawa,the facility is divided into a number of pre-characterized cells (suchas a cluster of racks) over which the response of various parameters,such as maximum temperature, are known based on key parameters. Thesystem uses built-in cell to cell interaction rules to predict overallthermal performance and to optimize equipment layout. While this systemmay offer some improvements in speed over a full CFD analysis, it islimited to the use of pre-characterized cells, and does not provideanalysis below the cell level. Also, the cells must be characterizedusing, for example, a CFD analysis or physical testing.

Programs and systems such as those described above provide idealizedresults for the cooling performance of a facility and often fail toaccount for situations which often occur in real life installations,which can dramatically affect the cooling performance of a data center.For example, in a facility using a raised floor, the absence of one ormore floor panels, or the misplacement of one or more perforated floorpanels can greatly affect the cooling performance of the data center andcause the actual performance to vary greatly from a calculated idealizedperformance. Further, the degradation in performance of one or more airconditioning units can drastically change airflow and coolingcharacteristics of a facility.

The inability to properly analyze the cooling performance of a facilitytypically causes a data center designer to over design the facility froma cooling perspective, which results in the facility to be moreexpensive and less efficient.

In existing data centers, it is often desirable to replace equipmentwith upgraded equipment and/or add new equipment to existing enclosuresin the facility. Several tools exist which enable a manager of a datacenter to monitor power usage in a facility. These tools include theInfraStruxure™ Manager product available from American Power ConversionCorporation of West Kingston, R.I.

With the increasing cooling and power requirements of computerequipment, it is desirable for a data center manager to determine ifthere is adequate power and cooling available in the facility before newor replacement equipment may be added. Typically, a data center managermay know, or can determine, if the total cooling capacity of the datacenter is sufficient for the total power draw. However, hot spots in afacility may develop, particularly where high power density equipment isused, and it may not be enough to merely analyze cooling capacity at thefacility level. To attempt to identify hot spots, a data center managermay resort to manual temperature measurements throughout a facility andtry to implement fixes to correct the hot spots. Such fixes may involvea rearrangement or replacement of perforated floor panels, arearrangement of enclosures, and/or adding additional cooling capacity.In any case, these fixes are typically done on a trial and error basis,and while some hot spots may be eliminated, the fixes may cause otherhot spots to arise due to a redirection of the cooling air in thefacility. This trial and error approach can lead to unexpected failuresof equipment, which is unacceptable in critical data centers. To avoidsuch failures, data center managers typically over design facilities andfail to use facilities to their full capacity.

SUMMARY OF INVENTION

Embodiments of the invention provide systems and methods for determiningdata center cooling and power requirements and for monitoringperformance of cooling and power systems in data centers. At least oneembodiment provides a system and method that enables a data centeroperator to determine available power and cooling at specific areas andenclosures in a data center to assist in locating new equipment in thedata center.

A first aspect is directed to a method that includes determining coolingcapacity of a number of equipment enclosures in a data center,determining cooling requirements of each of the number of equipmentenclosures, and providing an indication of remaining cooling capacityfor each of the number of equipment enclosures.

The method may further include developing a floor plan model of the datacenter, wherein the floor plan model includes a floor plan thatindicates location of each of the number of equipment enclosures in thedata center, and for each of the number of equipment enclosures,displaying on the floor plan, the indication of remaining coolingcapacity. The indication of remaining cooling capacity may include anindication of additional power that can be drawn by each of the numberof equipment enclosures based on the remaining cooling capacity.Determining cooling capacity may include calculating a predicted coolingcapacity based on the floor plan model. Determining cooling capacity mayinclude measuring airflow at a first plurality of locations in thefacility to obtain a measured cooling capacity. Determining coolingcapacity may include measuring air temperature at a second plurality oflocations in the facility. At least one of the first plurality oflocations and the second plurality of locations includes at least oneair vent of a raised floor. The method may further include comparingpredicted cooling capacity with measured cooling capacity to obtain acomparison result and providing an indication when the comparison resultis greater than a threshold. The method may further include adjustingthe predicted cooling capacity based on measured airflow. The method mayfurther include determining placement of new equipment in an equipmentenclosure in the data center by comparing power ratings of the newequipment with cooling capacity of the equipment enclosure. The methodmay further include, for each of the number of equipment enclosures,determining electrical power capacity and remaining electrical poweravailability, and displaying on the floor plan remaining electricalpower availability. In the method, determining remaining electricalpower availability may include measuring at least one parameter ofelectrical power provided to at least one of the number of equipmentenclosures. In the method, determining cooling capacity of an equipmentenclosure may include estimating available cooling air at the equipmentenclosure using a weighted summation of available airflows from aplurality of airflow sources, the weights used in the weighted summationmay decrease with distance from the equipment enclosure to each of theairflow sources, and the weights may be based on mechanicalcharacteristics of the plurality of airflow sources. The method mayfurther include determining available airflow of at least one of theplurality of airflow devices using at least one of specifications of theat least one of the plurality of airflow devices and measured data forthe at least one of the plurality of airflow devices in the data center.The method may further include determining available airflow of at leasta second one of the plurality of airflow devices based on the measureddata for the at least one of the plurality of airflow devices. In themethod, determining cooling capacity may include using superposition tocombine airflows. In the method, determining airflow into and out ofeach of a plurality of sides of each control volume may includecomputing airflows using equations based on at least one of conservationof mass and conservation of momentum. Further, determining airflow intoand out of each of a plurality of sides of each control volume mayinclude determining airflows using empirical rules derived from CFD,physical measurement, or any other means.

A second aspect of the invention is directed to a system for managing adata center. The system includes at least one input to receive datarelated to equipment and equipment enclosures and to receive datarelated to cooling characteristics of the data center, a controlleroperatively coupled to the input and configured to determine coolingcapacity of each equipment enclosure, and at least one outputoperatively coupled to the controller that provides an indication ofremaining cooling capacity for each of the equipment enclosures.

The system may further include an output device coupled to the at leastone output, wherein the system is configured to display a floor plan ofat least a portion of the data center indicating location of at leastone of the equipment enclosures in the data center and indicating theremaining cooling capacity for the at least one of the equipmentenclosures. The output device may be configured to include an indicationof additional power that can be drawn by the at least one of theequipment enclosures. The system may further include at least oneairflow monitor operatively coupled to the controller to provide datarelated to at least one airflow in the data center. The system mayfurther include at least one air monitor operatively coupled to thecontroller to provide data related to air temperature at a location inthe data center. The controller may be configured to compare a predictedcooling capacity with a measured cooling capacity to obtain a comparisonresult and to provide an indication when the comparison result isgreater than a threshold. The system may further include at least onepower monitor operatively coupled to the controller, and at least oneairflow controller operatively coupled to the controller and responsiveto signals from the controller to alter cooling airflow in the datacenter. The system may further include at least one power controlleroperatively coupled to the controller and responsive to signals from thecontroller to alter at least one characteristic of power in the datacenter.

A third aspect of the invention is directed to a system for managing adata center. The system includes at least one input to receive datarelated to equipment and equipment enclosures and to receive datarelated to cooling characteristics of the data center, and means,coupled to the at least one input, for determining remaining coolingcapacity for each of the number of equipment enclosures and providing anindication of remaining cooling capacity.

The system may further include means for providing an indication ofadditional power that can be drawn by each of the equipment enclosuresand means for updating the indication of remaining cooling capacitybased on measured airflows in the data center. The system may stillfurther include means for determining placement of equipment in the datacenter based on remaining cooling capacity, and means for estimatingavailable cooling air of at least one of the equipment enclosures usinga weighted summation of available airflows from a plurality of airflowsources.

Another aspect of the invention is directed to a computer-implementedmethod of managing power and cooling capacity of a data center. Themethod includes receiving data center parameters, determining anequipment layout in the data center, the equipment layout identifying alocation of each of a plurality of devices in the data center, based onthe location, determining available power and available cooling in thedata center for each of the plurality of devices, comparing theavailable power and available cooling with power requirements andcooling requirements of each of the plurality of devices to obtain acooling comparison result and a power comparison result for each of theplurality of devices.

The method may further include comparing each of the comparison coolingresults and the power cooling results with thresholds and providing atleast one recommendation for correcting an out of tolerance condition.The at least one recommendation may include adding an in-row coolingunit in a row of equipment of the data center along wit ha proposedlocation for the in-row cooling unit. The method may further includedetermining at least one of stranded cooling capacity and stranded powercapacity in the data center and providing recommendations for reducingat least one of the stranded power capacity and the stranded coolingcapacity in the data center. The method may further include displaying afloor plan model of the data center, wherein the floor plan modelincludes a floor plan that indicates a location of each of the pluralityof devices in the data center, and for each of the plurality of devices,displaying on the floor plan an indication of remaining coolingcapacity. The method may include displaying on the floor plan anindication of remaining power capacity for each of the plurality ofdevices. The method may still further include setting a redundancy levelfor at least some of the plurality of devices, and wherein the availablepower and available cooling are determined to meet the redundancy level.In the method, the act of determining the equipment layout may includearranging the plurality of devices in at least two substantiallyparallel rows with a hot aisle formed between the two rows, and themethod may further include conducting a cooling analysis by analyzingairflows in the hot aisle. The method may include selecting at least onein-row cooling unit to be placed in the layout in one of the at leasttwo substantially parallel rows. At least one of the plurality ofdevices is an equipment rack, and the method may include determining acapture index for the at least one in-row cooling unit and the equipmentrack. The method may further include on a display screen, simultaneouslydisplaying a first floor plan model of the data center and a secondfloor plan model of the data center, wherein the first floor plan modelincludes at least a partial view of the data center and the second floorplan model includes a full view of the data center. The second floorplan model may include an indication of a portion of the data centerthat is shown in the first floor plan model. The method may furtherinclude displaying a three dimensional view of at least a portion of thedata center. The method may include selecting a camera location for usein the data center and displaying a three dimensional view of a viewingarea of the camera. The method may further include selecting a subset ofthe plurality of devices and defining a power zone for each device ofthe subset of the plurality of devices, wherein each power zone includesat least one UPS. In the method, each of the plurality of devices may bean equipment rack, and the method may further include selectingcomponents for each of the plurality of devices from a displayed list ofcomponents. The method may further include determining operational powercosts and operational cooling costs for a subset of the plurality ofdevices, and the operational power costs and operational cooling costsmay be determined in terms of kilowatts. The method may further includetransferring an electronic file containing data for the equipment layoutfrom a design system to a management system. The method may also includedisplaying measured data for the data center on a display of a layout ofthe data center.

Another aspect of the invention is directed to a computer-implementedmethod for designing a layout of a data center. The method includesreceiving information from a user regarding parameters of the datacenter, determining an equipment layout for the data center, theequipment layout identifying a location of each of a plurality ofdevices in the data center, and on a display screen, simultaneouslydisplaying a first floor plan model of the data center and a secondfloor plan model of the data center, wherein the first floor plan modelincludes at least a partial view of the data center and the second floorplan model includes a full view of the data center.

In the method, the second floor plan model may include an indication ofa portion of the data center that is shown in the first floor planmodel. The method may further include determining available power andavailable cooling in the data center for each of the plurality ofdevices, and comparing the available power and available cooling withpower requirements and cooling requirements of each of the plurality ofdevices to obtain a cooling comparison result and a power comparisonresult for each of the plurality of devices. The method may include, foreach of the plurality of devices, displaying on the first floor planmodel an indication of remaining cooling capacity, and displaying on thefirst floor plan model an indication of remaining power capacity foreach of the plurality of devices.

Another aspect of the invention is directed to a system for use with adata center. The system includes an input to receive instructions from auser, an output to provide output data to a display device, and acontroller coupled to the input and to the output and configured todetermine an equipment layout of a data center, the equipment layoutidentifying a location of each of a plurality of devices in the datacenter. The controller is further configured to, based on the location,determine available power and available cooling in the data center foreach of the plurality of devices, and compare the available power andavailable cooling with power requirements and cooling requirements ofeach of the plurality of devices to obtain a cooling comparison resultand a power comparison result for each of the plurality of devices.

In the system, the controller may be configured to compare each of thecomparison cooling results and the power cooling results withthresholds, and based on at least one of the cooling comparison resultand the power comparison result, provide at least one recommendation forcorrecting an out of tolerance condition. The at least onerecommendation may include adding an in-row cooling unit in a row ofequipment of the data center, and the at least one recommendation mayinclude a proposed location for the in-row cooling unit. The controllermay be configured to determine at least one of stranded cooling capacityand stranded power capacity in the data center, and to providerecommendations for reducing at least one of the stranded power capacityand the stranded cooling capacity in the data center. The controller maybe further configured to provide data to the output for displaying afloor plan model of the data center, wherein the floor plan modelincludes a floor plan that indicates a location of each of the pluralityof devices in the data center, and provide data to the output fordisplaying on the floor plan an indication of remaining coolingcapacity. The controller may be further configured to provide data tothe output for displaying on the floor plan an indication of remainingpower capacity for each of the plurality of devices, and the controllermay be configured to determine the available power and available coolingbased on a user-selected redundancy level. The controller may beconfigured to arrange the plurality of devices in at least twosubstantially parallel rows with a hot aisle formed between the tworows, and to conduct a cooling analysis by analyzing airflows in the hotaisle. The controller may be configured to select at least one in-rowcooling unit to be placed in the layout in one of the at least twosubstantially parallel rows. At least one of the plurality of devicesmay be an equipment rack, and the controller may be configured todetermine a capture index for the at least one in-row cooling unit andthe equipment rack. The controller may be configured to provide data tothe output to simultaneously display a first floor plan model of thedata center and a second floor plan model of the data center, whereinthe first floor plan model includes at least a partial view of the datacenter and the second floor plan model includes a full view of the datacenter. The second floor plan model may also include an indication of aportion of the data center that is shown in the first floor plan model.The controller may be configured to provide data to the output todisplay a three dimensional view of at least a portion of the datacenter, and to provide data to the output to display a three dimensionalview of a viewing area of a camera to be located in the data center. Thecontroller may be further configured to select a subset of the pluralityof devices and define a power zone for each device of the subset of theplurality of devices, wherein each power zone includes at least one UPS.The system may further include a database module containing data forcomponents to be loaded into the plurality of devices, and thecontroller may be configured to provide data to the output fordisplaying a list of the components. The controller may be configured todetermine operational power costs and operational cooling costs for asubset of the plurality of devices, and the operational power costs andoperational cooling costs may be determined in terms of kilowatts. Thecontroller may also be configured to provide an output electronic filecontaining data for the equipment layout. The controller may also beconfigured to, based on at least one of the cooling comparison resultand the power comparison result, provide at least one recommendation forplacement of rack mount equipment.

Another aspect of the invention is directed to a computer-implementedmethod for designing a layout of a data center. The method includesreceiving information from a user regarding parameters of the datacenter, determining an equipment layout for the data center, theequipment layout identifying a location of each of a plurality ofdevices in the data center, including identifying a row location foreach of a plurality of equipment racks with a first subset of theplurality of equipment racks being included in a first row; and on adisplay screen, displaying a rack view of the data center showing afront view of each of the plurality of equipment racks of the firstsubset in the first row.

In the method, displaying a rack view may include displaying a frontview of a second subset of the plurality of equipment racks of a secondrow along with the front view of the first subset of the plurality ofequipment racks. In the method, displaying a rack view may includedisplaying a front view of a second subset of the plurality of equipmentracks of a second row along with the front view of the first subset ofthe plurality of equipment racks. The first row may include additionalequipment, with the additional equipment not included in the rack view.The method may further include simultaneously displaying on the displayscreen the rack view and a full room view of the equipment layout forthe data center. The method may also include, in response to selectionof a selected equipment rack in the full room view by a user, displayingthe selected equipment rack in the rack view, and displaying theselected equipment rack in the rack view may include displaying a frontview of the selected equipment rack.

Another aspect of the invention is directed to a computer-implementedmethod for evaluating the cooling performance of a cluster of equipmentracks in a data center, wherein the cluster of equipment racks includesat least a first row of racks and a second row of racks separated by acool aisle, with each of the equipment racks being configured to drawcooling air from the cool aisle. The method includes obtaining at leastone of power data and airflow data for each of the equipment racks,obtaining cool airflow data for cool air supplied to the cool aisle froma source of cool air, and conducting an analysis of airflows in the coolaisle to determine a recirculation index for at least one of theequipment racks, wherein the recirculation index is indicative of aquantity of recirculated air included in an input airflow of the atleast one equipment rack.

In the method, the recirculation index may be equal to a ratio ofrecirculated air to total air in the input airflow of the at least oneequipment rack, and the method may further include determining arecirculation index for each of the equipment racks. In the method, theact of obtaining cool airflow data may include obtaining cool airflowdata for an in-row cooling unit included in the cluster of racks. In themethod, the act of obtaining cool airflow data may include obtainingcool airflow data of at least one perforated tile included in the coolaisle. In the method, the act of conducting an analysis may includedefining a plurality of control volumes in the cool aisle, and themethod may further include determining airflows in the cool aisle bydetermining airflow into and out of at least one of the control volumes.The method may further include comparing the recirculation index foreach of the plurality of equipment enclosures with a threshold. Themethod may further include determining a cooling capacity for each ofthe equipment enclosures based on the recirculation index for each ofthe equipment enclosures, and displaying the cooling capacity for eachof the equipment enclosures along with a representation of a data centercontaining the cluster. In the method, the act of conducting an analysismay include assigning different chemical concentration identifiers tothe airflows for at least two of the plurality of equipment racks. Inthe method, the act of conducting an analysis may include importingempirical data and determining end of aisle airflows using the empiricaldata. In the method, the act of conducting an analysis may includedetermining cool aisle airflows in isolation from the data center toobtain isolated results, and combining the isolated results with theempirical data.

Another aspect of the invention is directed to a computer-implementedmethod for evaluating the cooling performance of a cluster of equipmentracks in a data center, wherein the cluster of equipment racks includesat least a first row of racks and a second row of racks separated by ahot aisle, with each of the equipment racks being configured to exhaustair into the hot aisle. The method includes obtaining at least one ofpower data and airflow data for each of the equipment racks, obtainingairflow data for at least one air removal unit contained in one of thefirst row of equipment racks and the second row of equipment racks, andconducting an analysis of airflows in the hot aisle to determine acapture index for at least one of the equipment racks, wherein thecapture index is indicative of a fraction of air that is exhausted bythe at least one of the equipment racks and captured by the at least oneair removal unit.

In the method, the at least one air removal unit may include an in-rowcooling unit, and the capture index may be equal to a ratio of capturedair to total air exhausted by the at least one equipment rack. Themethod may further include determining a capture index for each of theequipment racks. In the method, the act of conducting an analysis mayinclude defining a plurality of control volumes in the hot aisle, andthe method may further include determining airflows in the hot aisle bydetermining airflow into and out of at least one of the control volumes.The method may further include comparing the capture index for each ofthe plurality of equipment enclosures with a threshold. The method mayinclude determining a cooling capacity for each of the equipmentenclosures based on the capture index for each of the equipmentenclosures, and displaying the cooling capacity for each of theequipment enclosures along with a representation of a data centercontaining the cluster. In the method, the act of conducting an analysismay include assigning different chemical concentration identifiers tothe airflows for at least two of the plurality of equipment racks. Inthe method, the act of conducting an analysis may include importingempirical data and determining end of aisle airflows using the empiricaldata. The act of conducting an analysis may include determining hotaisle airflows in isolation from the data center to obtain isolatedresults, and combining the isolated results with the empirical data. Inthe method, the act of conducting an analysis may include importingempirical rules, and determining the capture index using the empiricalrules, and the empirical rules may include coefficients for use indetermining at least one capture index.

Another aspect of the invention is directed to a computer-readablemedium encoded with instructions for execution on a computer system. Theinstructions, when executed, perform a method comprising acts ofobtaining at least one of power data and airflow data for a plurality ofequipment racks arranged in a cluster, wherein the cluster of equipmentracks includes at least a first row of racks and a second row of racksseparated by a cool aisle, with each of the equipment racks beingconfigured to draw cooling air from the cool aisle, obtaining coolairflow data for cool air supplied to the cool aisle from a source ofcool air, and conducting an analysis of airflows in the cool aisle todetermine a recirculation index for at least one of the equipment racks,wherein the recirculation index is indicative of a quantity ofrecirculated air included in an input airflow of the at least oneequipment rack.

The recirculation index may equal to a ratio of recirculated air tototal air in the input airflow of the at least one equipment rack, andthe acts may further include determining a recirculation index for eachof the equipment racks. The act of obtaining cool airflow data mayinclude obtaining cool airflow data for an in-row cooling unit includedin the cluster of racks. The act of obtaining cool airflow data mayinclude obtaining cool airflow data of at least one perforated tileincluded in the cool aisle. The act of conducting an analysis ma includedefining a plurality of control volumes in the cool aisle, and whereinthe method further includes determining airflows in the cool aisle bydetermining airflow into and out of at least one of the control volumes.The acts may further include comparing the recirculation index for eachof the plurality of equipment enclosures with a threshold, anddetermining a cooling capacity for each of the equipment enclosuresbased on the recirculation index for each of the equipment enclosures.The acts may further include displaying the cooling capacity for each ofthe equipment enclosures along with a representation of a data centercontaining the cluster. The act of conducting an analysis may includeassigning different chemical concentration identifiers to the airflowsfor at least two of the plurality of equipment racks. The act ofconducting an analysis may include importing empirical data anddetermining end of aisle airflows using the empirical data. The act ofconducting an analysis may include determining cool aisle airflows inisolation from the data center to obtain isolated results, and combiningthe isolated results with the empirical data.

Another aspect of the invention is directed to a computer-readablemedium encoded with instructions for execution on a computer system. Theinstructions when executed, perform a method comprising acts ofobtaining at least one of power data and airflow data for a plurality ofequipment racks arranged in a cluster, wherein the cluster of equipmentracks includes at least a first row of racks and a second row of racksseparated by a hot aisle, with each of the equipment racks beingconfigured to exhaust air into the hot aisle, obtaining airflow data forat least one air removal unit contained in one of the first row ofequipment racks and the second row of equipment racks, and conducting ananalysis of airflows in the hot aisle to determine a capture index forat least one of the equipment racks, wherein the capture index isindicative of a fraction of air that is exhausted by the at least one ofthe equipment racks and captured by the at least one air removal unit.

In the method, the at least one air removal unit may be an in-rowcooling unit, and the capture index may be equal to a ratio of capturedair to total air exhausted by the at least one equipment rack. The actsmay further include determining a capture index for each of theequipment racks. The act of conducting an analysis may include defininga plurality of control volumes, and wherein the method further includesdetermining airflows in the hot aisle by determining airflow into andout of at least one of the control volumes. The acts may further includecomparing the capture index for each of the plurality of equipmentenclosures with a threshold. The acts may further include determining acooling capacity for each of the equipment enclosures based on thecapture index for each of the equipment enclosures, and displaying thecooling capacity for each of the equipment enclosures along with arepresentation of a data center containing the cluster. The act ofconducting an analysis may include assigning different chemicalconcentration identifiers to the airflows for at least two of theplurality of equipment racks. The act of conducting an analysis mayinclude importing empirical data and determining end of aisle airflowsusing the empirical data. The act of conducting an analysis may includedetermining hot aisle airflows in isolation from the data center toobtain isolated results, and combining the isolated results with theempirical data. The act of conducting an analysis may include importingempirical rules, and determining the capture index using the empiricalrules. The empirical rules include coefficients for use in determiningat least one capture index.

BRIEF DESCRIPTION OF DRAWINGS

The accompanying drawings are not intended to be drawn to scale. In thedrawings, each identical or nearly identical component that isillustrated in various figures is represented by a like numeral. Forpurposes of clarity, not every component may be labeled in everydrawing. In the drawings:

FIG. 1 is a top view of a data center of the type with which embodimentsof the present invention may be used;

FIG. 2 is a side view of the data center of FIG. 1.

FIG. 3 is a functional block diagram of a system in accordance with oneembodiment of the present invention;

FIG. 4 is a flowchart of a process that may be implemented using thesystem of FIG. 3 in accordance with one embodiment of the invention;

FIG. 5 is a diagram showing facility information that can be displayedusing at least one embodiment of the invention;

FIGS. 5A and 5B are diagrams showing additional information that can bedisplayed using embodiments of the invention;

FIGS. 5C and 5D show graphical user interface screens that can be usedin some embodiments of the present invention;

FIG. 6 is a functional block diagram of a management system inaccordance with one embodiment of the invention;

FIG. 7 is a flow chart of a management process in accordance with oneembodiment of the invention;

FIG. 8 shows a perspective view of a cluster of racks whose coolingperformance can be analyzed using embodiments of the invention;

FIG. 9 shows a top view of a cluster of racks whose cooling performancecan be analyzed using a control volume analysis technique of at leastone embodiment;

FIG. 9A shows the cluster of racks of FIG. 9 along with staggeredcontrol volumes that may be used in the control volume analysistechnique;

FIG. 10 is a flow chart of a control volume analysis technique inaccordance with one embodiment of the invention;

FIG. 11 is a diagram demonstrating a principle of superposition used inone embodiment;

FIG. 12 is a graph used in determining airflows in one embodiment;

FIG. 13 is a diagram identifying airflows used with one analysis methodof one embodiment;

FIG. 14 is a flow chart of a process for determining a recirculationindex in one embodiment;

FIG. 15 is a schematic diagram showing a layout of equipment racks usedin an analysis in one embodiment to determine a capture index;

FIG. 16 is a flowchart of a process for determining a capture index inaccordance with one embodiment;

FIG. 17 is a functional block diagram of a computer system that may beused in embodiments of the invention; and

FIG. 18 is a functional block diagram of a storage system that may beused with the computer system of FIG. 17.

DETAILED DESCRIPTION

This invention is not limited in its application to the details ofconstruction and the arrangement of components set forth in thefollowing description or illustrated in the drawings. The invention iscapable of other embodiments and of being practiced or of being carriedout in various ways. Also, the phraseology and terminology used hereinis for the purpose of description and should not be regarded aslimiting. The use of “including,” “comprising,” or “having,”“containing”, “involving”, and variations thereof herein, is meant toencompass the items listed thereafter and equivalents thereof as well asadditional items.

Embodiments of the present invention may be used to design, manage andretrofit a data center, such as data center 100 which is shown in FIGS.1 and 2 with FIG. 1 showing a top view of the data center 100, and FIG.2 showing a side view of the data center 100. As discussed furtherbelow, the design of the layout of the data center 100, including powerand cooling considerations may be performed using systems and processesof embodiments of the present invention. Embodiments of the invention,however, are not limited for use with data centers like that shown inFIGS. 1 and 2 and may be used with other facilities that do not includeraised floors and may be used with facilities that house equipment otherthan computing equipment, including telecommunications facilities andother facilities. Further, embodiments of the invention may be used withraised floor and equipment layouts that are not neatly arranged in themanner shown in FIGS. 1 and 2. Embodiments of the present invention mayuse systems, devices and methods described in U.S. patent applicationSer. No. 10/038,106, filed Jan. 2, 2002, titled “Rack Power System andMethod,” incorporated herein in its entirety by reference.

The data center 100 includes rows of racks 102A, 102B, 102C and 102D,cooling units 104A and 104B, and a raised floor 106. Each of the rowsincludes racks 108, at least a number of which draw cool air from thefront of the rack and return warm air to the rear or top or rear and topof the rack. As understood by those skilled in the art, to optimizecooling performance in a data center, rows of racks are often positionedto create alternating cold aisles and hot aisles. In the configurationshown in FIGS. 1 and 2, aisles 110A, 110B and 110C are hot aisles andaisles 112A and 112B are cold aisles. To provide cooling to the racks,in front of each of the racks in the cold aisle, perforated floor tiles114 are used to provide cooling air from under the raised floor. In thedata center 100, in addition to the perforated floor tiles shown, theraised floor may include solid floor tiles. The cooling units 104A and104B are designed to provide cool air to the area under the raised floorand to receive return warm air from the space adjacent the ceiling ofthe data center. In other embodiments, in addition to or in place of thecooling units 104A and 104B, in-row cooling units, such as thoseavailable from American Power Conversion Corporation, may be used.Further, in at least one embodiment, half-rack in-row cooling units maybe used, as described in co-pending U.S. patent application Ser. No.11/335,901 entitled COOLING SYSTEM AND METHOD (Attorney Docket No.A2000-704819), by Neil Rasmussen et al., filed on Jan. 19, 2006, andincorporated herein by reference. As described in the referencedapplication, the half-rack, in-row units have a width of twelve inches,which is approximately half of that of a standard data center rack.

One embodiment of the invention, directed to a system and a method fordesigning, monitoring, and upgrading the equipment installed in a datacenter, such as data center 100, will now be described with reference toFIG. 3. FIG. 3 shows a functional block diagram of a design andmanagement system 200. Embodiments of the invention are not limited tothe functions provided by the functional blocks or the particulararrangement of the blocks. In addition, the functions provided by thesystem 200 need not be implemented on one computer system, but rathermay be implemented using a number of networked devices as describedfurther below that provide the functions described. Further, particularembodiments may have more or less functions and functional modules thanthose described below with reference to FIG. 3. In differentembodiments, the functions described with reference to FIG. 3 may beperformed on one processor or controller or may be distributed across anumber of different devices.

The system 200 includes an input module 202, a display module 204, abuilder module 206, a facility management module 208, an integrationmodule 210, and a database module 212. The input module 202 provides aninterface to allow users to enter data into the system 200. The inputmodule may include, for example, one of a number of known user inputdevices for computer systems, and in addition, in at least oneembodiment, electronic data regarding a facility and/or equipment to beloaded into a facility may be entered into the system through a networkinterface or using an electronic media storage reader.

The display module includes a display interface and may include agraphical display to display output data to a user. In addition, thedisplay module may include an interface for one or more printers thatprovide a hard copy of output data.

The builder module 206 includes routines for designing optimal layout ofequipment in a facility, determining power requirements and coolingrequirements for electronic enclosures, ensuring that the placement ofequipment, cooling units and power distribution branches in the facilityallow the power and cooling requirements to be met, and calculating foreach electronic enclosure the remaining power capacity and coolingcapacity available based on the layout of equipment in the facility.

The facility management module 208 is used by the system 200 afterequipment is installed in the facility. The management module includesroutines to monitor power and cooling characteristics of equipment in afacility. The management module may be coupled, either directly orthrough one or more networks, to measurement devices and control devicesthroughout the facility.

The integration module 210 is the main module in the system andcoordinates flow of data in the system to perform methods of embodimentsof the present invention.

The database module is used to store data regarding various devices thatmay be used in a data center, such as servers, uninterruptible powersupplies, air conditioning units, racks and any other equipment. Thedata stored may include physical parameters (i.e., dimensions) as wellas power and cooling consumption data, and in the case of power suppliesand air conditioning units may include cooling and power outputcharacteristics. As described below, the database module may be used inembodiments of the invention to provide a complete bill of materials(BOM) for a completed design. In one embodiment, a centralizedweb-accessible database server may be used to store equipmentinformation and warnings and error messages, allowing easy access to theinformation for editing.

A flow chart of a method 300 in accordance with one embodiment that maybe performed using the system 200 will now be described with referenceto FIG. 4. Initially, at stage 302 of the method 300, informationregarding the facility is loaded into the system. The informationincludes, for example, dimensions of the facility, locations of doors,support columns, parameters of available power, cooling capabilities ofthe facility, whether a raised floor or drop ceiling is in use, andcharacteristics of any such floor and roof.

In stage 304 of the method, information regarding equipment to beinstalled in the facility is entered. The information includes, forexample, the number of racks of equipment, maximum power draw for eachof the racks, dimensions of the racks, and cooling requirements for theracks. The need for backup power sources and multiple power sources forequipment and or racks may also be entered at stage 304. In oneembodiment, characteristics of individual pieces of equipment that areto be loaded into racks may also be entered. Also, the weight ofequipment (including equipment loaded into racks) may be used to ensurethat the weight of the installed equipment is within any facilityconstraints. These characteristics may include, in addition to power andcooling requirements, the amount of rack space that the equipment needsto occupy. In one embodiment, the database module 212 containsinformation regarding a number of devices, such as uninterruptible powersupplies, equipment racks, cooling units, generator systems, electricalrouting devices, including cables, and servers and other computerequipment. In this embodiment, when a particular model number of adevice is entered, characteristics of the device are retrieved from thedatabase module. Equipment related to fire protection and security mayalso be included in the design. Further, in at least one version, allequipment and components within equipment racks may include RFID tags,which can be used by systems of the invention to track location ofequipment and racks.

Once all of the information is entered into the system, at stage 306,the system in one embodiment determines a layout for the equipment inthe facility, taking into account the power and cooling requirements ofthe equipment as well as other characteristics of the equipment thatwere entered at stage 304 or retrieved from the database module. Inanother embodiment, the user may create the layout graphically, addingracks and other equipment where desired, and in this embodiment, thesystem will provide feedback during the layout process, disallowing somechoices and making intelligent suggestions. These rules may include, forexample: a standard alternating hot aisle/cold aisle layout must bespecified, the plenum must be greater than some minimum value, the totalroom cooling capacity must exceed total room cooling load, aisles mustbe wide enough for access purposes and to meet building codes, distancebetween PDU and IT racks served by the PDU must not exceed some maximumvalue, PDU must be located immediately adjacent to a UPS, where a cableladder spans an aisle, the aisle cannot exceed a maximum width, etc.

Next, at stage 308, a cooling analysis is conducted to determine if thedesign provides adequate cooling for each of the racks and the equipmentinstalled in the racks. As described further below, in differentembodiments of the present invention one of a number of differentmethods may be used to conduct the cooling analysis. In one embodiment,if the results of the cooling analysis indicate that one or more devicesand/or racks are not receiving adequate cool air, then the procedure mayreturn to stage 306 to change the layout of the equipment based onfeedback provided from the analysis conducted at stage 308.

At the completion of the cooling analysis, at stage 310, a room model isdisplayed showing the locations of the equipment in the facility. Theroom model may include, for each equipment rack, information regardingthe total power and cooling being consumed as well as an indication oftotal available power and cooling to the rack. In one embodiment actualpower and cooling data may be displayed, while in other embodimentscolors may be used, either alone or in combination with data, to displaydifferent levels of power and cooling availability. For example, if arack is operating with sufficient cooling air with a margin above athreshold, the rack may be indicated in green on the display, if thecooling air availability is closer to the threshold, the rack may beindicated in yellow, and if the rack does not have sufficient coolingair it may be indicated in red. Still further, the results of theanalysis may indicate that adequate power and/or cooling is beingprovided for equipment, but that specified redundancy levels are notbeing met, either at the room level, a row level, or at a specificequipment rack. Specific details regarding the room model is describedfurther below with reference to FIGS. 5 and 5A to 5D.

At decision block 312, a determination may be made by, for example, afacility designer as to whether the layout generated in stage 310 issatisfactory. The determination may be based on additional criteria ofimportance to the designer that was not included during the design ofthe original layout. For example, it may be desirable to have certainracks near each other or to have certain racks isolated from oneanother. At stage 314, additional criteria or other feedback can beprovided and the process then returns to stages 306 and 308 where theroom model can be refined. Stages 306 to 312 may be repeated until asatisfactory model is achieved at stage 312. In at least one embodiment,at the completion of the design stage, a bill of materials is generatedand may be used to provide the cost of the equipment to be installed inthe facility and may also be used to generate a sales order for theequipment, providing a simple solution for ordering all equipmentassociated with a new data center. Further, CAD drawings and electronicfiles that capture the designed layout may also be generated.

At stage 316, the equipment is installed in the facility according tothe layout generated at stages 306 to 314. In one embodiment,measurement equipment to measure cooling characteristics and powercharacteristics may be installed with the equipment. The measurementequipment is described further below, and may include, for example,devices for measuring power, airflow and temperature at variouslocations in the facility and within equipment racks located in thefacility.

At stage 318 of the process 300, power and cooling parameters aremeasured using the measurement equipment. Additional temperaturemeasurements may also be provided by devices, such as servers, that havethe capability to detect internal temperatures. The parameters measuredmay be used continuously by the management module of the system 200 todetect error conditions and to monitor trends that may lead to an errorcondition. Further, in the process 300, the measured parameters can becompared with predicted parameters calculated during the design processin stages 306 and 308. For example, in one embodiment, the airflowthrough a perforated floor tile of a raised floor is used to determinethe available cooling air of a rack located adjacent the floor tile. Theairflow through the perforated tile may be determined in stage 308 usingone of a number of computational methods that are described furtherbelow, or the airflow may be determined using data from related physicalmeasurements or simulations. Once the equipment is installed in thefacility, the perforated floor tile may be instrumented to measure theactual airflow through the tile. The actual measured value may then becompared with the calculated value at stage 320. If the two differ bymore than a predetermined threshold, then an indication or warning maybe provided and the calculations conducted in stage 308 may be conductedonce again at stage 322 using measured values in place of calculatedvalues as appropriate to obtain updated parameters.

After stage 322, the model of the facility described above withreference to stage 310 may be displayed with values of power and coolingavailability and consumption updated to reflect any differences betweenmeasured parameters and calculated parameters. Any out of toleranceconditions (for either cooling or power) may be indicated on the displayusing, for example, a color coded scheme as described above. In oneembodiment, a user may be provided with a number of available options tocorrect an out of tolerance condition. The options may include upgradingor adding facility equipment (i.e., an air conditioning unit or anuninterruptible power supply) or may include moving equipment and/orracks. Stages 318 to 322 of the process may be performed continuously aspart of a management system of the data facility.

In one embodiment of the invention, stages 302 to 314 of the process 300are implemented using a build-out system accessible by a user over theInternet. In this embodiment, the user provides the requestedinformation, and the build-out system provides the processing describedabove, provides outputs to the user over the Internet, and storesresults locally. After the equipment has been installed in the facility,the management system 500 (described below) may access the build-outsystem to download information related to the equipment. In addition,when a retrofit of the facility is to occur, the management system maycontact the build-out system to coordinate the design of the retrofit.In at least one embodiment, electronic files may be imported/exportedbetween the systems to provide a complete transfer of all informationrelated to a data center's design.

FIG. 5 shows an example of a display of a room model that may begenerated using the system 200 and the process 300 and shown on acomputer display. The room model shown in FIG. 5 is essentially the datacenter 100 previously discussed above with reference to FIGS. 1 and 2,however, in FIG. 5, additional data related to the power and coolingconsumption and capacity of each rack may be included in aninformational block, such as informational blocks 120A and 120B shown ontwo of the racks 108A and 108B in FIG. 5. Informational blocks may beincluded on each rack, or on less than all racks, for example, by row,zone, or cluster.

FIGS. 5A and 5B show enlarged views of respectively racks 108A and 108Bthat may also be shown on a computer display of systems of embodimentsof the invention. In the views of FIGS. 5A and 5B, specific informationregarding the racks is included in the informational block. In theembodiment shown, the information in the block includes a rackidentifier 122, a rack type 124, power capacity 126, power usage 128,cooling capacity 130, cooling usage 132 and contents of the rack 134. Inother embodiments, information for each rack may be included in tabularform on a graphical display showing the room layout.

The rack identifier 122 includes a row number and a rack number,however, in other embodiments, the rack identifier may also include anindicator of the type of rack, membership of the rack to a particularrow, zone or cluster, manufacturer of the rack, as well as otherinformation. The rack type 124 identifies the particular type of rack,i.e., server rack, router rack or telecommunications rack. The powercapacity 126 indicates the maximum power capacity of the rack, and thepower usage indicator 128 indicates the percentage of maximum capacityat which the rack is expected to operate. In different embodiments, thepower usage indicator may be calculated based on manufacturer supplieddata for equipment contained in the rack and/or based on actual powermeasurements of equipment. The power capacity for a rack, in at leastone embodiment, is determined based on limitations of devices and/orpower cables that supply power to the rack, such as circuit breakers,UPS's or any other devices. The contents of the rack 134 includes a listof the equipment contained in the rack and may include an indication ofremaining space in the rack displayed, for example, in terms of rackunits, which are typically referred to as “U” with 1U equal to 1.75inches. Details regarding the equipment in the rack, includingoperational status and network addresses, such as an IP address for adevice may also be included.

The cooling capacity indicator 130 and cooling usage indicator 132identify respectively the quantity of cooling air available to the rackand the percentage of that cooling air that is being used by equipmentin the rack. In other embodiments power and cooling usage may beindicated using various types of graphical gauges, such as a bar graph,that indicates power and cooling usage and capacity. In the embodimentshown in FIGS. 5A and 5B, the cooling capacity is shown in terms ofkilowatts (kW). As known to those skilled in the art, for typical datacenter applications, many equipment racks typically requireapproximately 160 cfm (cubic feet per minute) of cooling air perkilowatt of power used by the rack. All the power consumed by computingtype devices is typically converted to heat, such that the requiredcooling (in terms of kW) for a rack can be assumed to be equal to thepower consumption of the rack. Accordingly, in one embodiment, thecooling usage indicator is equal to the power consumed by the rack. Inother embodiments, depending on the type of equipment that is installedin the racks, the cooling required by a rack may not be equal to thatconsumed by the rack and may be calculated based on manufacturer's datafor the equipment, based on test results, or in any other manner.

The cooling capacity of an equipment rack is determined based on anumber of different factors. For example, for a raised-floorenvironment, these factors may include: location of the rack in thefacility, proximity of perforated tiles to the rack, the amount andtemperature of cooling air provided through any such perforated tile,the physical or geometric layout of the racks and building, and thecooling requirements of other equipment in the facility located near therack. The cooling capacity of one rack in a facility may be affected bythe cooling usage of nearby racks, and accordingly, in one embodiment,the cooling capacity of a rack is adjusted when the cooling usage of anearby rack is changed. In at least one embodiment of the presentinvention, calculations for determining cooling capacity of a rack arebased in part on the ability of one rack to borrow cooling air availableto adjacent racks. Particular methods for determining cooling capacityfor racks in embodiments of the present invention are described furtherbelow. In one embodiment, when the cooling usage of one rack is changed,the cooling capacity of that rack, and all racks located near thechanged rack is recalculated.

In embodiments of the present invention, during the design as well asduring the management of a datacenter, the true available capacity of adata center can be determined at the rack level, at the row level and atthe room level. In determining available capacity (including unusedcapacity), both cooling and power capacity are used, and the trueavailable capacity can be determined using the lower of the power andthe cooling capacity. In situations where the power capacity is notequal to the cooling capacity, then the excess power capacity or coolingcapacity can be considered wasted or stranded capacity that can not beused in the present design. In embodiments of the present invention, thestranded capacity can be determined at the rack level and can be totaledto determine stranded capacity at the row level and at the room level.Recommendations are provided for reducing stranded capacity during thedesign phase as well as during the management phase. The recommendationsmay include reducing capacity of power and cooling (thereby reducingoperational costs) or adding cooling capacity or power capacityappropriately to reduce the stranded capacity. Warnings may be generatedwhen the stranded capacity is greater than preset thresholds, and inaddition, recommendations for more optimal locations of equipment,including power and cooling equipment, may also be provided to minimizethe amount of stranded capacity. Further, costs of the stranded capacitymay be calculated.

In management systems and methods of embodiments of the invention, asdescribed above, power and cooling capacity and availability may bemonitored in real time. In one version, changes to the availability rate(or the utilization rate) are monitored and based on these changes, thegrowth rate of a data center may be determined, and predictions of dateswhen additional capacity will be required can be provided. The abilityto monitor capacity and predict future capacity needs allows data centeroperators to control costs and plan for upcoming expenses. Further,determinations may be made as to the additional expenses that will beincurred if new equipment is added. The total cost (for example perkilowatt) can also be determined during the design phase or duringoperation.

In embodiments of the invention described herein, datacenter layouts maybe designed to provide specific redundancy levels (i.e., n, n+1, or 2n)for both the power design and the cooling design. In data centers in thepast, additional room cooling units are typically provided to includesome redundancy in a datacenter, such that an overall cooling capacityof the datacenter can be maintained, even when one or more of the roomcooling units fails or must be powered down to conduct maintenance. Oneproblem with these past solutions is that the cooling redundancy isdesigned at the room level and not the rack level, and while overallcooling capacity may meet redundancy requirements, cooling at the racklevel may not meet the desired redundancy requirements. In embodimentsdescribed herein, the ability to provide accurate airflow data at therack level allows true cooling redundancy to be designed into asolution.

As discussed above, graphical user interfaces may be used withembodiments of the present invention to assist in the design andmanagement of data centers. Particular user interface screens used inone embodiment to design a layout in a data center will now be describedfurther with reference to FIGS. 5C and 5D. FIG. 5C shows a floor editorinterface 402 used in one embodiment to layout equipment in a datacenter, while FIG. 5D shows a rack editor interface 404 used in oneembodiment to provide further details of the contents of equipment inthe data center. In one embodiment of a data center design system,tutorials are provided for a user to assist the user by providing bestpractice design guidelines. The tutorials may be accessed by a user asdesired or may be configured to be displayed as a user is taking aparticular action.

The floor editor interface includes a main menu 403, a tool bar 406, aconfiguration box 408, a generic components box 410, a floor layout box412, a status box 414 a full-image viewing box 416, and an unplacedequipment box 418. The main menu 403 provides a drop-down menu in aformat known to those skilled in the art, and allows a user to performvarious functions, including the ability to “undo” and/or “redo” changesthat are made to the layout. The tool bar 406 provides short hand accessto functions of the design system, and in one embodiment includes afloor editor button 406A and a rack editor button 406B. Activation ofthe floor editor button results in the display of the screen shown inFIG. 5C, while activation of the rack editor button results in displayof the screen shown in FIG. 5D.

The floor editor box 412 shows the layout of equipment in a data centerbeing designed and provides text that identifies the equipment containedin the layout. A room perimeter 412A shows the exterior walls of theroom along with dimensions of the room that can be set by a user. In oneembodiment, when a new design is started, the user is presented with ascreen showing a number of basic room configurations that can beselected. Further, the walls of the room can be moved by a user byselecting one of buttons 412B, and additional buttons can be added whereneeded to expand or shrink an area of the room. In one embodiment, theroom size may be changed as equipment is added to the layout. Three rows412C, 412D and 412E are outlined in the room shown in FIG. 5C. In otherembodiments, more or less rows may be included. As shown in FIG. 5C, therows are configured in a manner to provide alternating hot and coldaisles. Row 412D includes three equipment racks (identified by “R”), twohalf-rack cooling units (identified by “C”) a UPS (“U”) and a powerdistribution unit (“P”). Row 412E includes one rack, and row 412C aspresently configured does not include any equipment. During the designphase additional equipment may be added to each of the rows. The roomalso includes an automatic transfer switch (ATS) 412G and a coolingdistribution unit (CDU) 412F. Hatched areas are shown on the displayaround the ATS and CDU to indicate that these areas should be kept clearof equipment. Each piece of equipment in the room may includeidentifiers that indicate the type of rack as well as the rack'slocation in the room and the power source for the rack. Further, asdiscussed above, each rack may include information regarding power andcooling use and availability. Still further, text may be provided oneach row to indicate total power and cooling information for each row.

The configuration box 408 includes eight configuration options fordesigning a data center. A room properties configuration option, whenselected, allows a user to identify physical, power, and cooling valuesthat affect the data center design as a whole including dimensions ofthe room, aisle widths and total anticipated power density for the datacenter. Power redundancy requirements (i.e. N, N+1, 2N), coolingredundancy requirements and runtime requirements for UPS systems mayalso be set. The number of data troughs that will be used and locationof power distribution and cooling line distribution (i.e. overhead orunder a raised floor) can also be configured. In one embodiment, onlyin-row cooling is provided, however, in other embodiments other types ofcooling solutions may be used as well. In at least one embodiment,individual rows may be rotated to different angles in the data center.Further, while only one room is shown in FIG. 5C, at least oneembodiment allows a data center to include multiple rooms.

An add rack configuration option in the configurations box 408 is usedto add equipment racks to the data center design. When this option isselected, a user is presented with choices of various types of racks toadd to the data center. When racks are selected, an indicator isprovided in the unplaced equipment box 418, indicating that the racksstill need to be placed into the room layout.

An add in-row cooling option in the configuration box is used to addin-row cooling units to the data center design. When this option isselected, a user is presented with various types of cooling units thatcan be added in the rows. As with equipment racks, when a cooling unitis selected, an indicator is provided in the unplaced equipment box 418,indicating that the cooling unit still needs to be placed in the roomlayout.

A power zone option in the configuration box is used to identify andselect PDU's and UPS's and to indicate which equipment will be poweredfrom the UPS's and PDU's. Characteristics of the PDU's and UPS's mayalso be selected. Once selected, an indicator is provided in theunplaced equipment box 418 for the UPS's and PDU's. In one embodiment,multiple racks may be included in a selection on the layout to identifythe equipment that belongs to a particular power zone. In still anotherembodiment, after selection of equipment and UPS's and PDU's, anautomatic power zone option may be implemented in which the systemmatches equipment power requirements (i.e., redundancy levels, voltages,phasing) to those of the UPS's and PDU's and assigns power zonesautomatically and determines lengths of power cables that are needed topower equipment from the assigned PDU. In automatically determiningpower zones, the system may also identify the need for additional UPS'sand PDU's.

A power generation option in the configuration box 408 is used toidentify and select an automatic transfer switch (ATS) and generator.Again, once these are selected, an indicator is provided in the unplacedequipment box 418.

An emergency power off option in the configuration box is used to selectan emergency power off (EPO) solution for the data center design, andonce selected, an indicator for the EPO solution will be added in theunplaced equipment box.

A management option in the configuration box 408 allows a data centermanager, such as the InfraStruxure Manager discussed above, to be added.In one embodiment, when selecting the manager, a rack location for themanager is also selected.

A service option in the configuration box 408 allows a user to select alevel of service to be provided to the data center by a data centerservices organization.

Other configuration options may include a row planning configurator thatallows a user to plan how many racks a row can support by defining thepower and cooling settings for the row, prior to placing equipment in arow. In one embodiment, the row planning configurator may provide anestimate on the number of racks that can be supported based on the powercomponents and in-row cooling units contained in the row. In oneembodiment, the row planning configurator may provide a complete layoutbased on design best practices.

The generic components box 410 includes a number of icons to designatepre-existing equipment in a data center. The components may be selectedand “dragged” into position in the layout. In one embodiment, thegeneric components include blocks and gaps. The gaps can be used toidentify areas over which cables and conduits can be routed (i.e. awalkway), while the blocks are used to identify areas over which cablesand conduits can not be routed (i.e. a column). Once dragged onto thelayout, the blocks and gaps can be sized appropriately.

As discussed above, when equipment is selected for use in the datacenter, an icon appears in the unplaced equipment box 418. To place theequipment in the layout, the icon is selected and dragged into theappropriate location in the layout. In one embodiment, when adding anin-row cooling unit, the icon for the cooling unit can be placed betweentwo adjacent racks and released, and the racks will then move in the rowto allow the cooling unit to be inserted between the racks. Further, inone embodiment, a snap-to feature is employed to align equipmentproperly in rows and along walls, and in addition, rows and equipmentmay be aligned along and “snapped to” floor tiles when, for example, araised floor is in use. Using this feature, a user does not need toprecisely align objects.

The full-image viewing box 416 provides a “bird's eye” view of thelayout contained in the floor layout box 412. In one embodiment, thezoom button on the tool bar can be used to enlarge the view of the datacenter layout in the floor layout box 412. When the view is enlarged,the entire layout may not appear in the floor layout box. The full-imagebox 416 still displays the full image of the layout for the user. In oneembodiment, when the full layout does not appear in the floor layoutbox, an overlay is used in the full-image box to indicate on thefull-image, the portion of the layout that is displayed in the floorlayout box. In one embodiment, when the full image is not displayed inthe floor layout box 412, the overlay may be selected and dragged withinthe full-image viewing box to select which part of the layout isdisplayed in the floor layout box.

The status box 414 is used to display warnings, errors and otherconditions to the user. The warnings may vary in severity and mayinclude indications that design guidelines are being violated and mayalso include more severe warnings indicating that power and coolingcapacities have been exceeded. In one embodiment, when the status boxindicates that there is an error or warning associated with a particularpiece of equipment in the layout, the piece of equipment may behighlighted with a color such as red or yellow. In at least oneembodiment, when an error or warning occurs, guidelines for correctingthe error or warning are provided by either selecting a highlightedpiece of equipment or the error or warning message directly.

The rack editor interface 404 will now be described further withreference to FIG. 5D. The rack editor interface includes the tool bar406, the status box 414 and the full-image viewing box 416 discussedabove. Further, the rack editor interface 404 also includes a rackeditor box, a product catalog box 422 and a rack content section 424.

The rack editor box 420 shows the front face of each of the equipmentracks in the data center layout with the racks being arranged by row. InFIG. 5, two rows of racks 420A and 420B are shown. As shown in FIG. 5,in one embodiment, only the equipment racks are shown in the rack editorbox. When a particular rack is selected in the rack editor box, then thecontents of the rack appear in the rack content box 424, and componentsmay be added to the selected rack. Racks may be selected in the rackeditor box or may also be selected in the full-image view box 416. Whena rack is selected in the full-image view box, then the image in therack editor box will shift, if necessary, to provide a view thatincludes the selected rack.

The product catalog box 422 provides a comprehensive listing ofcomponents that may be used in equipment racks in data centers. The usermay select equipment to be included in each rack, and as equipment isselected, it is included in the rack content box 424. The list mayinclude only equipment of a particular manufacturer or may includeequipment from several manufacturers. In one embodiment, all necessaryhardware and cabling associated with rack equipment may be selected fromthe product catalog box.

In another embodiment, in addition to the graphical user interfacescreens shown above, a three-dimensional option is available allowing auser to view the design of a data center in 3D. In one embodiment, adesign system includes software code programmed in Java that is used togenerate 3D models that are rendered via OpenGL to allow for hardwareacceleration. Further, 3D models may be exported from the design systemto CAD tools such as AutoCAD, available from AutoDesk of San Rafael,Calif. As described above, security cameras can be implemented intodatacenters designed using embodiments of the present invention. In oneversion that has 3D viewing capabilities, security cameras may beincluded in the design and the 3D view may be used to view a simulationof a camera's view after installation. In one embodiment, networkedcameras and other security monitoring devices available from NetbotzCorporation of Austin, Tex. may be used.

As discussed above, with reference to the process shown in FIG. 4, thesystem 200, and other systems of the present invention, may be used aspart of a data center management system. The management system mayinclude the system 200 described above with the management modulecontaining routines to perform management functions, or in otherembodiments, the management functions may be performed by a designatedmanager controller contained in the data center and implemented, forexample, in a computer server located in one of the racks of equipmentand accessible by a user using a management console.

FIG. 6 shows a block diagram of a management system 500 that may be usedin embodiments of the present invention. The management system includesthe manager controller 502, the manager console 504, power measurementdevices 506, and airflow and temperature measurement devices 508. Inaddition, in one embodiment, the management system may include powercontrol devices 520 to control application of power to one or moreindividual devices or racks contained within a data center, and thesystem may include airflow controllers 521 to control the airflow orsupply temperature of an air conditioning unit or to control, forexample, perforated tile dampers. As discussed above, the managementsystem may also include one or more security devices 523, includingsecurity cameras. The devices of the management system 500 may bedirectly coupled to the manager controller or may be coupled to themanager controller using a network 522 that may be a dedicated network,may include the Internet, or may include a LAN or WAN contained in thedata center. The manager controller may communicate with one or moreservers 524 to obtain information from and control operation of theservers.

In one embodiment, the management controller 502 may be implemented atleast in part using an Infrastruxure® Manager available from AmericanPower Conversion Corporation (APC) of West Kingston, R.I., and devicesmay be coupled to the manager using, for example a controller areanetwork (CAN) bus. The power controllers and airflow controllers may beimplemented using available known devices that monitor and/or controlpower and airflow in facilities. Further, in at least one embodiment,the management system 500 may include systems and methods for monitoringand controlling power as described in U.S. Pat. No. 6,721,672 toSpitaels et al, which is incorporated by reference herein. Further, inat least one embodiment that uses in-row cooling devices, the managementcontroller may communicate with the cooling units to control the unitsto ensure that adequate cooling at specified redundancy levels is beingmet. Further details regarding the control of in-row cooling units thatcan be used with embodiments of the invention are discussed in copendingU.S. patent application Ser. No. 11/335,901 discussed above and filed onJan. 19, 2006.

One aspect of the present invention, which will now be described, isdirected to a retrofit system and method that is particularly useful foradding new equipment to a facility. The addition of new equipment mayinclude adding equipment to existing racks or may include addingequipment racks to a facility. The retrofit system may be a standalonecomputer system configured to perform processes described herein, or inone embodiment, the retrofit system is implemented using the system 200described above. Specifically, the builder module 206 of the system 200may include routines to assist in retrofitting a data center. A process600 for using the system 200 (or some other system) to retrofit orupgrade a data center will now be described with reference to FIG. 7,which shows a flow chart of the process 600.

In a first stage 602 of the process 600, data related to a presentconfiguration of a data center to be retrofitted is provided to thebuilder module. The data related to the present configuration mayinclude the data displayed in the room model of FIG. 5 along withadditional data that was generated during design of the data center.Further, in one embodiment, the data related to the presentconfiguration may include data generated during an initial design asupdated by actual measurements conducted in a facility. For example, thecooling capacity of individual racks may be calculated in an initialdesign and then updated by the management module once the system isinstalled and operating. Cooling capacity data may be updated based onactual measurements of airflow from, for example, perforated floortiles, while the original data may have been calculated based onpredicted airflows.

Information related to the retrofit is then provided in stage 604. Theinformation related to the retrofit may include information similar tothat input at stage 304 of process 300 described above, such as type ofequipment, characteristics of equipment, number of racks, as well asother information. In addition, a user may designate one or more desiredlocations in the data center for the installation of new equipment. Forexample, a user may desire to add five additional servers to the datacenter, where the servers are similar to and have a related function toexisting servers in the data center. The user may choose one or morepreferred locations based on power specifications, coolingspecifications and physical dimensions of the servers, and based onpower capacity, cooling capacity and contents of existing racksdisplayed on a floor model of the data center. In addition, the user mayindicate whether it is acceptable to move existing equipment toaccommodate the installation of new equipment.

At stage 606, an updated layout for the data center is generated andcooling and power calculations are performed at stage 608 on the updatedlayout in the manner discussed above at stage 308 of process 300. If theuser has designated specific locations for equipment in the data center,the layout may first be determined using these locations, and ifproblems arise as a result of the desired layout (i.e., lack of coolingfor a rack), then the user will be able to note any such problems oncethe layout is displayed and can then choose to change the layout. If aparticular layout is not designated by a user, then the system 200 willdetermine the layout in the manner discussed above with respect to stage306 of process 300.

At stage 610, an updated floor model is displayed (for example, in themanner shown in FIGS. 5C and 5D), and at stage 612, a user can reviewthe model and either provide feedback (stage 614) or indicate that thedesign is satisfactory. Once the floor model has been approved by auser, the retrofit design process is complete, and the equipment may beinstalled and parameters of the data center may be measured and updatedin the manner described above in stages 318 to 322 of process 300 usingfor example a data center management system.

In the processes 300 and 600 described above, design and analysis stagesare performed after all data is entered as part of an initial design ora retrofit of a facility. In another embodiment, analysis is performedreal-time, and user displays are updated as the user enters data intothe system.

In embodiments of the present invention, using the processes describedabove, data center operators are able to determine, in essentiallyreal-time, whether additional equipment may be added to a data centerand may also determine locations for the equipment, where both power andcooling requirements of the equipment may be met. Further, reports canbe generated that indicate to a user or data center manager how muchcapacity is available for each row, for each rack and for the facilityin its entirety. Still further, as discussed above, in determiningoverall capacity, systems and methods are able to identify strandedcapacity, and provide suggestions for reducing the stranded capacity.

In the processes and systems described above, cooling calculations for adata center and for equipment in the data center are performed as partof the design process for the data center, during operation of the datacenter, and during an upgrade or retrofit of the data center. Inembodiments of the present invention, in determining equipment layoutand performing cooling calculations, initial information oncharacteristics of the facility itself are identified to determine ifthere is sufficient cooling at the facility level. These characteristicsinclude, for example, whether a raised floor or drop ceiling is used asan air plenum, the location and characteristics of air conditioningunits (including in-row cooling units), dimensions of the room that areto house the data center, and total power draw of the data center. Basedon this information, an initial determination may be made as to whetherthere is sufficient cooling provided by the air conditioning units forthe expected power draw in the room, and if not, a recommendation may bemade for additional air conditioning units. For some facilities, desiredredundancy and operating margins may be included in this determination.

Once the determination has been made that there is sufficient cooling atthe facility level, an analysis is conducted to determine if there isadequate cooling at each rack in the facility and/or at individualpieces of equipment. In at least one embodiment, the cooling capacity ofa rack may be determined by increasing the power level of the rack todetermine at what additional power level the airflow to the rack becomesinadequate. This can be performed individually for each of the racks ina data center. In different embodiments of the present invention, one ormore of a number of different methods may be used to perform the coolingcalculations. These methods include, but are not limited to, acomputational fluid dynamics (CFD) analysis, a Coarse-Grid CFD analysis(designated as CGCFD), a control volume analysis (designated as CVA),and an analysis based on empirical rules and/or borrowing concepts.Further, in some embodiments, a combination of two or more of the abovemethods may be used to conduct portions of an overall analysis. In oneembodiment, the principle of superposition is used to combine results ofportions of an analysis. In particular, in many applications theairflows may be considered to be ideal, where an ideal airflow isinviscid, incompressible, irrotational without any other forces, such asbuoyancy. With such an ideal airflow, a complex application can bereduced to a number of less complex applications, analysis of the lesscomplex applications can be performed using one of the methods describedherein, and superposition can be used to combine the results of each ofthe less complex applications to obtain analysis results for the complexapplication.

A computational fluid dynamics analysis can be used in one embodiment inassociation with the design and retrofit of a data center to providedetailed results of the cooling performance of a facility, includingdetermining the availability of adequate cooling air at racks andindividual pieces of equipment in the facility and determining coolingcapacity for each rack. The techniques for implementing a CFD analysisof a data center are known. A CFD analysis must typically be performedby someone particularly skilled in the art, typically requires detailedinformation regarding the facility and the layout of equipment in thefacility, and depending on the complexity of the analysis conducted, andthe computing equipment used to conduct the analysis, may take hours ordays to run one iteration of the analysis.

In another embodiment, an improved technique for conducting the coolinganalysis is used. The improved technique has been developed based oncomputational fluid dynamics techniques. The improved technique isreferred to herein as a Coarse-Grid CFD or simply CGCFD. In a typicalCFD analysis, a data center to be analyzed is typically divided intonon-uniform cells in the range of one to eight inches on a side. In atleast one embodiment, in the CGCFD analysis, a Cartesian grid system ofcells that are one foot cubes are used. The use of uniform one footcells typically reduces the number of cells used in the calculationsfrom a traditional CFD analysis by at least an order of magnitude.Further, uniform grid cells generally make the CFD analysis faster andmore reliable relative to a comparable non-uniform cell analysis.Further, other techniques are employed in the CGCFD analysis to improvethe computational efficiency of the analysis. These techniques include:the use of simple turbulence models, initializing the analysis with dataobtained from the results from a prior similar solution, using twodimensional or partial two dimensional representations when possible tosimplify calculations, and tailoring a CGCFD routine for a specificapplication. The use of two dimensional representations may be used, forexample, in a raised floor or ceiling plenum, where pressure gradientsin the depth direction can be neglected in the computations.

The tailoring of a CGCFD routine can be used in embodiments of thepresent invention to significantly improve computational efficiency andimprove robustness (for example, so the tool can be made to workreliably in an autonomous way) of the total analysis, and multipletailored routines may be used in combination to produce a completeanalysis. For example, a first CGCFD routine may be tailored for usewith different raised floor configurations to determine the outputairflow at each perforated floor tile of a raised floor in a datacenter, and a second CGCFD routine may be tailored for use with acluster of racks that include two rows of racks with a cold aislebetween the rows. The first CGCFD routine may be run to determine theoutput air at perforated tiles in the cold aisle of the rack cluster,and the second CGCFD routine may use the results of the first routine todetermine the airflows and temperatures at inlets and outlets of theracks. The second routine may be run a number of times to account forall of the clusters of racks located in a data center. As equipment ismoved, and as different configurations are established within a clusterto optimize cooling performance, the second routine may be run to obtainnew cooling data without the need to repeat the first routine, as theairflows from the perforated tiles generally would not change. In somecases, for perforated floor tiles having a large percentage of open area(for example, greater than 50%), it may be desirable to repeat the firstroutine as air flows may change based on room configuration.

Embodiments of the invention that utilize the CGCFD approach to conductthe analyses of a data center provide advantages over embodiments thatutilize a traditional CFD approach. These advantages includecomputational efficiency and simplification of use. Iterations ofcooling calculations using the CGCFD approach may be conducted in amatter of seconds or minutes versus hours or days with a full CFDanalysis. Further, the CGCFD routines may be designed to operate with alimited set of input variables, allowing a less-skilled user to conductanalyses using the CGCFD approach. For example, for a CGCFD routine thatis tailored to analyze only the raised floor plenum, the input variablesmay be limited to the height of the floor, location and type ofperforated tiles, length and width of the floor, and the locations andcharacteristics of air conditioning units that provide cooling air tothe raised floor. For a CGCFD routine that is tailored to conduct ananalysis on a cluster of racks, the input data may be limited to airflowper tile (could be obtained automatically from the output of a separateCGCFD routine or using other methods), the number of racks in thecluster, the power draw of each of the racks, and room environmentaldetails including the temperature of the surrounding environment,ceiling height, the presence of nearby walls, etc. The output data for acluster of racks may include the input temperature at each server, orother piece of equipment in a rack. In other embodiments, the outputdata may simply be a measure of the amount of warm recirculated airdrawn into each rack. The data may be expressed as an absolute number(e.g. in terms of cfm) or expressed as a fraction of the total airconsumed by the rack. A system, such as system 200 described above, mayuse the output data to determine if the cooling performance of thecluster is satisfactory.

In another embodiment, another improved technique for conducting thecooling analysis is used. The improved technique is referred to hereinas a control volume analysis or simply CVA. The control volume analysismay be used in conjunction with a CFD analysis and/or a CGCFD analysis,or may be used as a stand alone process. The CVA technique is similar insome aspects to the CGCFD technique described above, however, furthersimplification of the analysis process is provided. As will be discussedbelow, the CVA technique is a computationally efficient technique thatis particularly effective for computing the three-dimensional airflow,pressure and temperature distributions in the cold aisle of a raisedfloor data center. However, the CVA technique is not limited in its useto this application and may be used for other applications as well. TheCVA technique can provide output data in essentially real-time, allowinga user to try various locations for equipment as part of an initialdesign or a retrofit and obtain cooling data for the different locationsin real-time.

The CVA technique will be described with reference to FIG. 8, whichshows a subsection 700 of a data center. The subsection of the datacenter includes a cluster of racks that includes a first row of racks702, and a second row of racks 704, which are located on a raised floorand separated by two rows of perforated tiles 706, 708.

In data centers that have clusters of racks arranged like those in FIG.8, it is not uncommon for undesirable hot spots to develop even thoughthe total supply of cool air to the cluster should be sufficient to meetthe needs of the racks. For example, if the airflow rate through one ormore perforated tiles is too great, a rack may be unable to capture allof the tile airflow and some of the cooling air may escape the coldaisle. Racks will generally draw their required air, and in thissituation, if a rack can not capture cool air, it may draw hot exhaustair over the top of the rack creating a hot spot. Further, due to widelyvarying cooling airflow requirements, racks may compete with one anotherfor cooling airflow. In particular, a high-power rack may borrowunderutilized air from an adjacent rack, or in some cases from a rackseparated by several tiles. With several racks contained in a cluster,with each having different cooling needs, the airflow patterns andtemperature distribution in the cold aisle are complex functions. TheCVA technique can be used to simplify the solutions to these complexfunctions.

In the CVA analysis for the rack cluster of FIG. 8, the airflow andtemperature analysis is conducted on the volume of air contained in thecold aisle, between the racks, from the perforated tiles up to a heightequal to the top height of the racks. The volume of air is divided intoa number of control volumes equal to the number of racks in the cluster.Each control volume is defined as the volume above one of the perforatedtiles extending from the perforated tile to the top of the racks. Thecontrol volume analysis includes determining for each control volume,the airflow through each of the six faces of the control volume. Oncethe airflows are known, temperatures and mass species concentrations canbe determined for each of the control volumes. In the CVA technique, thetemperature analysis can be decoupled from the airflow analysis because,as discussed above, buoyancy forces in the control volumes can beignored. Similarly, mass species concentrations are not coupled to theairflow solution and may also be computed separately if desired in orderto compute the fraction of recirculated air ingested by each rack.

In conducting a CVA analysis in the embodiment described herein, thereare several initial assumptions made to simplify the analysis. In otherembodiments, the analysis could be changed if these assumptions wouldnot apply. The first assumption is that airflow across each face of acontrol volume (and therefore into the front face of a rack) isconsidered uniform. Resulting airflow and temperature values effectivelyrepresent an average of the airflow and temperature at each face.

The second assumption is that buoyancy forces within each control volumeare negligible. Unless a significant hot spot develops, then there isinsufficient heating of the air in a cold aisle to substantially affectairflow patterns, and even if some heating occurs, any buoyancy effectsare small compared to the momentum of the airflow from typicalperforated tiles.

The third initial assumption is that viscosity and turbulence withineach control volume are negligible. In the control volumes, air isintroduced through the perforated tiles and is pulled into the racks.The air is not required to change direction rapidly and there is no flowof air parallel to solid objects. Accordingly, viscosity and turbulencemay be ignored and the competition of forces driving the airflow reducesto an interplay between pressure and momentum.

The CVA analysis may be further simplified by dividing a cluster ofracks into slices of two racks separated by two perforated tiles 718A,718B. FIG. 9 shows a cluster of six racks 710 that can be divided intothree two-rack slices 712, 714, 716. The nomenclature used foridentifying racks, and airflows in FIG. 9 is defined in Table 1 below,along with other variables that will be used herein in the descriptionof the CVA technique. TABLE 1 A_(s), A_(t) Control volume side andperforated tile area C₁, C₂ Dimensionless empirical constants in the yand x-momentum equations C Species Concentration CV Control volume NNumber of 2-rack slices in cluster {circumflex over (n)} Outward unitnormal vector PA_(i), PB_(i) Pressure in control volume above perforatedtiles A_(i) and B_(i) P_(amb) Ambient reference pressure M_(L), M_(R),M_(T) z-direction momentum flux terms through left, right, and topsurfaces of staggered CV at slice i TA_(i), TB_(i) Temperature incontrol volume above perforated tiles A_(i) and B_(i) Q_(t) Airflow ratethrough each perforated tile QA_(i), QB_(i) Airflow rate through racksA_(i) and B_(i) QAx_(i), QBx_(i) Airflow rates in the x-direction aboveperforated tiles A_(i) and B_(i) Qz_(i) Airflow rates in the z-directionabove perforated tiles between tiles A_(i) and B_(i) QAtop_(i), Airflowrates in the y-direction above perforated tiles A_(i) and QBtop_(i)B_(i) at top-of-rack height {right arrow over (V)} Velocity Vector αLinear relaxation factor ρ Density of air

At the start of the CVA analysis, the rack airflows QA_(i), QB_(i) andthe tile airflows are known. The tile airflows can be estimated based onthe mean perforated tile airflow for the entire facility or determinedusing a CFD analysis, a CGCFD analysis, physical measurement, or usingone of a number of known programs. The rack airflows can be determinedbased on characteristics of equipment installed in the rack. In oneembodiment, each rack airflow is determined on the basis of the powerusage of the rack and using the relationship of 160 cfm per kilowatt ofpower as discussed above. To determine the airflow patterns, allairflows QAx_(i), QBx_(i), Qz_(i), QAtop_(i), and QBtop_(i), andpressures PA_(i), and PB_(i) are computed based on the principle ofconservation of mass and momentum. To perform this computation, a totalof 7n−2 unknowns (5n−2 internal airflows plus 2n pressures) may bedetermined using a total of 7n−2 equations, where n is the number of2-rack slices (or length of cluster expressed in tile or rack widths).Optionally, an energy balance or mass species balance can then be usedto compute the 2n temperatures or 2n species concentrations based on theairflows.

In one embodiment, rather than solve all equations simultaneously, asemi-simultaneous approach is taken. In this embodiment, the fiveunknowns for each two-rack slices of a rack cluster, airflows Qz_(i),QAtop_(i), and QBtop_(i), and pressures PA_(i), and PB_(i), are firstdetermined simultaneously. During these initial computations, eachtwo-rack slice is considered in isolation, which is the equivalent ofhaving the ends of the slices blocked, such that QAx_(i) and QBx_(i) areequal to zero. After an initial sweep through each two-rack slice iscomplete, the side-to-side airflows (QAx_(i), QBx_(i),) can be computedbased on the calculated pressures within each control volume. Theside-to-side airflows affect the pressures, and after computing the sideto side airflows, a second computation of the airflows and pressures isconducted for each of the two-rack slices. This process is repeateduntil there are no significant changes in the computed variables. Onceall airflows are known, all temperatures or mass species concentrationscan be computed directly without the need to do multiple iterations.

The fundamental equations used to compute the unknowns described above,assuming steady state, incompressible and inviscid fluid flow rely onconservation of mass (m), conservation of momentum (M) conservation ofenergy (e) and conservation of species concentration (C), and can bewritten as follows: $\begin{matrix}{{\int_{A}^{\quad}{( {\overset{arrow}{V} \cdot \hat{n}} )\quad{\mathbb{d}A}}} = 0} & (m) \\{{\int_{A}^{\quad}{( {\rho{\overset{arrow}{V} \cdot \hat{n}}} )\overset{arrow}{V}\quad{\mathbb{d}A}}} = {- {\int_{A\quad}^{\quad}{p\hat{n}\quad{\mathbb{d}A}}}}} & (M) \\{{\int_{A}^{\quad}{{T( {\overset{arrow}{V} \cdot \hat{n}} )}\quad{\mathbb{d}A}}} = 0} & (e) \\{{\int_{A}^{\quad}{{C( {\overset{arrow}{V} \cdot \hat{n}} )}\quad{\mathbb{d}A}}} = 0} & (C)\end{matrix}$

Applying the conservation of mass equation (m) to the two-rack slicesfor the conditions described above results in the following equations:Q _(t) +QA _(i) +QAx _(i−1) =Qz _(i) +QAx _(i) +QAtop_(i)  (1)Q _(t) +Qz _(i) +QBx _(i−1) =QB _(i) +QBx _(i) +QBtop_(i)  (2)

Where QA_(i) is always negative based on the sign convention of FIG. 9.

As will now be described, staggered control volumes are used toformulate the z-momentum equations. Three staggered control volumes730A, 730B and 730C are shown in FIG. 9A. The number of staggeredcontrol volumes is equal to the number of 2-rack slices. The staggeredcontrol volumes are the same size as the main control volumes, but areshifted so that they are located midway between opposing racks. Thestaggered control volumes allow pressure to be considered more easilyfor each face which is normal to the z-direction. If the originalcontrol volumes are used, then each control volume would have one facecoplanar with a rack inlet, which is a face over which the pressure isnot known and need not be known in the calculations. Applying theconservation of momentum equation (M) in the z-direction to thestaggered control volume in slice i results in the following equation:PA _(i) −PB _(i)=(ρ/(4 A _(s) ²)) {(Qz _(i) +QB _(i))²−(QA _(i) +Qz_(i))² }+M _(L) +M _(R) +M _(T)  (3)

In equation (3), the first term on the right hand side of equation (3)is generally dominant, as it accounts for the effect of rack airflowrates on control volume pressures. M_(L), M_(R) and M_(T) account forlosses or gains in z-momentum through the sides and the top of thecontrol volume.

Using an “upwind” estimate for incoming/outgoing z-momentum and assumingthat the velocity of the air in the z-direction is negligible above theracks, M_(L), M_(R) and M_(T) are determined using the equations ofTable 2 below TABLE 2 IF THEN ELSE QAx_(i−1) + QBx_(i−1) ≧ 0 M_(L) =−(ρ/(2 A_(s) ²)) M_(L) = −(ρ/(2 A_(s) ²)) (QAx_(i−1) + QBx_(i−1))Qz_(i−1) (QAx_(i−1) + QBx_(i−1)) Qz_(i) QAx_(i) + QBx_(i) ≧ 0 M_(R) =(ρ/(2 A_(s) ²)) M_(R) = (ρ/(2 A_(s) ²)) (QAx_(i) + QBx_(i)) Qz_(i)(QAx_(i) + QBx_(i)) Qz_(i+1) QAtop_(i) + QBtop_(i) ≧ 0 M_(T) = (ρ/(4A_(s) ²)) M_(T) = 0 (QAtop_(i) + QBtop_(i)) Qz_(i)

The relationship between Y-momentum and pressure may be written usingequations (4) and (5) as follows:PA _(i) −P _(amb)==(ρ/A _(t) ²){C1[Q _(t)+½(QAi+QAx _(i−1) −QAx _(i) −Qz_(i))]²−½QAtop_(i) ²}  (4)PB _(i) −P _(amb)==(ρ/A _(t) ²){C1[Q _(t)+½(Qzi+QBx _(i−1) −QBx _(i) −QB_(i))]²−½QBtop_(i) ²}  (5)

In one embodiment, equations (1) through (5) are solved simultaneouslyfor each 2-rack slice of a cluster sequentially using the process 800shown in FIG. 10. In the first stage 802 of process 800, the userdefines Q_(T) (the airflow through the perforated tiles), the number of2-rack slices in the cluster, and the power draw of each of the racks.As discussed above, Q_(T) may be estimated as the mean perforated tileairflow rate for the entire facility or determined separately using, forexample, a CFD or CGCFD analysis or other analysis or physicalmeasurement. At stage 804, all airflow variables (except Q_(T) and therack inlet airflows) are initialized to zero. At stage 806, equations(1) through (5) are solved simultaneously for each slice. At decisionblock 808 a determination is made as to whether the equations have beensolved for all slices, and if not, stage 806 is repeated. Once theequations have been solved for all slices, then at stage 810, thex-direction airflow variables are updated based on the computedpressures in the control volumes, PA_(i) and PB_(i) as discussed below.At stage 812, a determination is made as to whether the computedpressures have changed by more than a predetermined threshold since theprevious iteration and if so, stages 806 to 812 are repeated. Once thereis no significant change in the computed variables, the process 800stops at stage 814, at which point the pressures and airflows for all ofthe control spaces have been determined.

In the process 800, at stage 810, new x-direction airflow values(QA_(xi) and QB_(xi)) are determined based on the assumption that thepressure drop between adjacent cells is proportional to the square ofthe airflow rate using the equations in Table 3. TABLE 3 IF THEN ELSEPA_(i) ≧ PA_(i+1) QAx_(i) = A_(s){(PA_(i) − PA_(i+1))/ QAx_(i) =−A_(s){(PA_(i+1) − PA_(i))/ (ρ C₂)}^(1/2) (ρ C₂)}^(1/2) PB_(i) ≧PB_(i+1) QBx_(i) = A_(s){(PB_(i) − PB_(i+1))/ QBx_(i) = −A_(s){(PB_(i+1)− PB_(i))/ (ρ C₂)}^(1/2) (ρ C₂)}^(1/2)

In one embodiment, because of non-linearities of the equations,adjustments to the x-direction airflow values at stage 810 are achievedgradually by introducing damping into the iterative process and updatingthe values of QAx_(i) and QBx_(i) using the following equations (6) and(7).QAx _(i) =αQAx _(i) ^(new)+(1−α)QAx _(i) ^(old)  (6)QBx _(i) =αQBx _(i) ^(new)+(1−α)QBx _(i) ^(old)  (7)

In equations (6) and (7), a is a linear relaxation factor. If α is setto zero, then no changes will occur from iteration to iteration. If α isset to 1, then there will be no damping introduced. For smaller valuesof α, more iterations will be required, however, the chances ofobtaining a stable solution increase. The particular optimum choice of αis problem specific, however, it has been found that values of α around0.05 work well in the process described above. Once the airflows arecomputed using the process above, temperatures and mass speciesconcentrations can be calculated, if desired. It should be noted thatcontrol volumes may be used to compute temperatures or concentrationsregardless of the method used to initially compute airflows.

The CVA technique described above can be conducted separately, one foreach cluster of racks in a facility to obtain a complete coolinganalysis of the facility. When a retrofit of a facility is to be done,the control volume analysis may be done for all clusters, or only forthose in the vicinity of any changes to the facility.

Three different methods, CFD, CGCFD and CVA, have been described abovefor determining cooling data in embodiments of the present invention todetermine placement of location of equipment in data centers. In stillanother embodiment, empirical rules are used either alone or incombination with one of the methods described above to determine properplacement of equipment and the adequacy of cooling air. The empiricalrules that are used may take a number of different forms, and programsincorporating the empirical rules may be updated as more data isgenerated to support the empirical rules. In one embodiment, empiricalrules are based, at least in part, on the ability of equipment racks toborrow unused capacity from surrounding neighbors. The amount that maybe borrowed may be limited to an allowable fraction (or weight) of theunused capacity and the particular allowable fraction may differdepending on a number of variables such as borrower-donor separationdistance, tile flow rate, and the total power draw of both the borrowerand the donor.

In one particular embodiment, the cooling air available to a given rackis computed based on a weighted summation of the available airflows fromairflow sources (i.e., supply devices, including in-row cooling units,or vents), net of airflows computed to be used by other racks, where theweights associated with the available airflows for a given rack decreasewith distance between the rack and the air supply devices or vents. Forexample, with reference to FIG. 9, the cooling air available to eachrack may initially be set equal to the cooling air supplied by theperforated tile in front of the rack, or to reflect possible losses, andprovide safety margin, the cooling air available may be set equal tosome amount (i.e. 90%) of the total air from the perforated tile. Thecooling load for each rack is then subtracted from the available air toprovide a net available cooling air figure for each perforated tile andto provide an initial indication of a lack of cooling air for anyequipment rack. For each equipment rack, the available cooling air isthen increased by assigning to each rack, a percentage of the netavailable cooling air from nearby perforated tiles. For example, thecooling air available may include 10% of the net available cooling airfrom a perforated tile associated with either an adjacent rack or a rackacross an aisle, and 5% of the net available cooling air from aperforated tile of a diagonal rack or a rack two positions over in arow. The particular percentages or weights used may be changed based onactual results or as a result of analyses conducted. The loads of eachrack may then be compared with the total available cooling air todetermine remaining cooling capacity and to identify any potentialproblem racks.

In at least one embodiment, empirical rules may be used in combinationwith superposition to analyze data centers and provide recommendedequipment layouts. Using superposition, complex problems may be brokendown into simpler problems that can then be solved using empiricalrules.

In one embodiment, empirical rules are established by initiallyperforming a number of CFD analyses on typical rack layouts, and theresults of these analyses are used to produce simple equations orlook-up tables that can be used in real-time to design layouts ofequipment. In such an analysis, the side-to-side airflows, such as thoseshown in FIG. 9 may be determined for each rack one at a time with onerack turned “on” and all other racks turned “off”. The airflows at theends of a cluster for a number of different configurations may also bedetermined using CFD. The airflows may be determined for a number ofdifferent air intake values for each rack and a number of differentvalues of air flow from the perforated tiles. The total air flows fordifferent configurations can then be determined in real-time usingsuperposition and the stored results. The airflows through the top (inor out) of the volume in front of each rack may then be determined basedon conservation of mass. In one embodiment, when the airflow into thetop of one of the volumes exceeds some percentage (i.e., 20%) of thetotal air flow into the rack associated with the volume, then anoverheating problem may exist requiring a design around. In otherembodiments, mass species concentration analyses may be used incombination with empirical rules to determine what percentage of thetotal air entering a control volume is recirculated air to determinewhen an overheating problem may exist.

In determining the airflows for each rack of a cluster, symmetry of theclusters can be used to reduce the number of CFD analyses that need tobe performed, and the control volumes discussed above with respect toFIG. 9 may be used to establish a reference grid for the analysis. Forexample, with reference to the cluster of racks 710 of FIG. 9, CFDanalyses need only be performed for Rack A_(i−1), and Rack A_(i), andthe results for each other rack may be determined based on the resultingairflows and the relative position of the racks. For example, theairflows in the cluster associated with Rack B_(i+1) are the same asthose associated with Rack A_(i−1) with the direction of some of theairflows changed for corresponding Rack A and Rack B inlet airflow andtile airflow rates.

In one example, which will now be described, the concepts of symmetryand superposition are used in conjunction with CFD analyses andempirical rules to provide a practical real-time solution fordetermining air flows in a cool aisle. Further, air flows are used todetermine a recirculation index (RI) for a row of racks, which can beused to identify potential “hot spots” in a data center. As discussedabove, one cooling objective in a data center is to manage the equipmentrack inlet air temperatures. The rack inlet air temperatures aredominated by the airflow patterns within the cold aisle and thetemperatures within and around the cold aisle. Air drawn in from outsidethe cold aisle is generally heated to some degree by the rack exhaustand will be hereafter referred to as “recirculated air”. While thetemperature of the recirculated air is highly application dependent, airthat passes directly from a perforated tile to a rack inlet will be verynear the supply temperature. Thus, good cooling performance can beachieved if all of the airflow ingested by a rack comes directly fromthe perforated tiles.

A cluster of racks, which receives its required cooling air exclusivelyfrom the perforated tiles within the cluster, represents an autonomousscalable unit from which a larger facility with predictable coolingperformance may be constructed. A reasonable requirement is therefore toensure that racks are adequately cooled by air originating from theracks own cold aisle. Conversely, it is acceptable for the rack toingest no more than a small fraction of recirculated air.

With the above in mind, the recirculation index (RI) is defined as thefraction of recirculated air ingested by the rack. An RI of 0% impliesthat all of the rack inlet air was drawn directly from the perforatedtiles while an RI of 100% implies that all of the rack inlet air wasdrawn from outside of the cold aisle. Note that a low RI is sufficientto guarantee cool inlet temperatures; however, a high RI does notguarantee excessively high inlet temperatures.

The concepts of control volumes, symmetry and superposition are used inthe present example to determine air flows and ultimately RI for a coldaisle. In using superposition, a sum of velocity potentials (or actualvelocity components or total airflows over a consistent area) ofsimpler, elemental flow solutions is used to obtain a new, compositeflow solutions. For example, assume we know the airflow patternassociated with only Rack A1 “on” subject to a particular tile airflowrate and we also know the airflow pattern with Rack B3 “on” subject tothe same perforated tile airflow rate. The relevant horizontal airflowcomponents can be added to obtain a solution, which approximates theairflow pattern resulting from Racks A1 and B3 both “on” simultaneously.The airflow pattern resulting from the superposition of the two separateairflow patterns is not exactly the same as the full solution—even foran ideal flow. Using superposition two solutions are added togetherwhich individually (and when added together) satisfy conservation ofmass criteria. The use of superposition does not guarantee that thecombined solution will be the unique solution and the difference is inthe boundary conditions. As an illustration of this, consider a 2-rackexample. In using superposition, the top airflow condition floats freelyas a constant-pressure boundary condition in all cases. In reality, theairflow pattern constructed from superposition ay not provide a perfectmatch to air velocity over the top surface of the cold aisle. Also, henone rack is off, an assumption is made that the face (inlet) of the rackis a symmetry boundary condition (which is consistent with an inviscidanalysis). This result creates the opportunity for some flow parallel tothe face of the rack, which would probably not exist when the rack isactually drawing air.

In the example, superposition is used to establish only the 3n−2internal horizontal airflows (n being equal to the length of the row interms of racks) while end-of-row horizontal airflows are computed basedon separate empirical correlations. Vertical airflow components arecomputed from a mass balance performed on each control volume. Thehorizontal airflows clearly depend on tile airflow. For example, a rackof a given airflow rate may be able to draw cooling air from a distanceof many tile-widths when the perforated tile airflow rate is very low.However, this range of influence is much less as the tile flow rate issubstantially increased. (As we know from the non-dimensional argument,the results would be identical if all airflows are scaled by the tileflow rate.) Therefore, the tile airflow rate is included in theanalysis; the floor tiles should be “turned on” in the CFD analysis usedto correlate airflow patterns. However, if the floor tiles are left “on”and the effect of each rack is considered individually, when the flowsfor each rack are summed, the sum would have more airflow leaving thetop of the cold aisle than in reality. The answer is to correlate onlythe horizontal airflows and then simply compute the correct airflow intoor out of the top of each control volume based on conservation of mass.

It is worth emphasizing that the use of the non-dimensional airflow and,in particular, superposition, simplifies the method. Without thesesimplifications, there would be many combinations of rack and tileairflows to evaluate and store empirically to cover a range of practicalapplications.

Based on the above discussion, the complete airflow solutions to anyrack layout of interest can be constructed using superposition.Elemental building-block airflow patterns are associated with each rackand each of the four end-of-row airflows are turned on individually asillustrated in FIG. 11 for the case of a 2-rack cluster 1002. It isimportant to stress that FIG. 11 illustrates which airflow boundaryconditions are turned on and off in each of the elemental airflowsolution to yield the total solution with all airflow boundaryconditions “on”. Each of arrows 1004 a to 1004 f represents one of theairflows. It is the airflow components internal to the cold aisle thatare actually being combined. There are, in general, a total of 2n+4elemental solutions for any layout, which makes up a complete airflowsolution. Obviously, fewer elemental solutions are required if someracks have zero airflow and the ends of the cold aisle are sealed (e.g.with doors).

The elemental airflows used with superposition may be determined in anymanner including physical testing. In the present example, CFD modelingfor the cold aisle is performed using the following boundary conditions:

-   -   Fixed velocity of air leaving the computational domain over the        area of a rack face for any rack which is “on”    -   Fixed velocity entering or leaving the domain over the area of        the end of the rows for any end-of-row flow “on”.    -   The top of the solution domain is “open” for air to enter or        exit to the surrounding environment held at constant pressure.    -   All other surfaces are “symmetry” surfaces.

As stated above, there are in general 2n+4 elemental solutions for eachrow length; 2n elemental solutions associated with each rack turned onplus four end-of-row elemental solutions. Each elemental solution coversa range of dimensionless flow rates so that any arbitrary, butpractical, rack or end airflow rate can be considered. So, the task isreduced to determining and storing the 3n−2 internal horizontal controlvolume airflows over an appropriate range of dimensionless airflowrates.

Because of the geometric symmetry of a cluster of racks, only the 3n−2internal airflows for approximately one quarter of the 2n+4 rack andend-of-row boundary conditions are considered and stored; n/2+1 boundaryconditions if n is even and (n+1)/2+1 if n is odd. The remaininginternal airflows are determined from an appropriate reinterpretation ofthe smaller data set by changing variable indices and signs. In additionto being efficient, this use of symmetry, forces the final output fromthe rack cooling performance tool to be perfectly symmetric. Each ofthese boundary conditions are driven individually through a range ofdimensionless airflow rates while keeping track of all of the “response”airflow rates internal to the cluster. The result can be summarized in aplot of “response” airflow rates; one plot for each elemental boundarycondition.

As an example, internal horizontal airflows associated with boundarycondition Rack A1 for an n=2 cluster are shown in FIG. 12. There are 4curves in FIG. 12 because there are 4 horizontal internal airflowsassociated with an n=2 cluster of racks. All of these curves can beconveniently approximated with a least-squares fit to a cubic polynomialof the generic formQ*=c ₁ (QRA ₁*)+c ₂ (QRA ₁*)² +C ₃ (QRA ₁*)³  (8)so that only the coefficients c₁, c₂, and C₃ must be stored for allairflows associated with all unique boundary conditions for all n's.Storing the “response” airflow as an equation offers the additionalbenefit compared to a simple look-up table in that results outside thedomain of FIG. 12 are automatically interpolated.

The process involved in compiling the curves in FIG. 12 and theconstants of Equation 8 is repeated for all unique boundary conditionsfor all n's considered. Determining all internal airflow correlations,for example, up to n=30 requires several hundreds of CFD runs.Therefore, in at least one embodiment, the process of converting the rawCFD data into the curve-fit constants of Equation 8 is automated. In atleast some examples above, the flow in the cold aisle is considered tobe ideal with no viscosity or turbulence. To verify this assumption,sample CFD cases were run with turbulence and viscosity included, andlittle difference was detected between models that included viscosityand turbulence and those that did not.

The discussion above describes a process for all internal cold-aisleairflows for any row length, perforated tile airflow, and rack airflowdistribution assuming that that the end airflow is known. A process forpredicting the end airflow will now be described.

Unlike the airflow within the cold aisle, the end airflow is stronglycoupled to the airflow in the surrounding room environment. Buoyancyforces can be significant; direct superposition of rack-induced airflowsmay not work well and the end airflows do not depend simply on thedimensionless rack airflow rates. The end airflow can still bedetermined using empirical correlations of CFD data; however, arelatively large number of CFD simulations typically should be performedin order to achieve reasonable accuracy over a useful range of actuallayouts. A comprehensive model for end airflow, which takes into accountdifferent geometric and thermal environments, may be included in otherembodiments. In one embodiment, described herein, a method includespredicting end airflow as a function of rack power and airflowdistribution for any row length and perforated tile flow rate whileassuming a fixed room environment. The example environment is large andfree of other racks or objects. Air is supplied at 60° F. and isexhausted uniformly over a 14 ft-high ceiling. As discussed above, underideal-flow conditions, we can expect air velocity at points near theracks to scale with the dimensionless rack inlet velocities. Further, asdiscussed above these “response” velocities vary nearly linearly withdimensionless rack flow rate (or velocity). It is, therefore, reasonableto estimate the dimensionless end airflows based on the followingexpression:QAx ₀ *=a ₀ +a _(A1) QRA ₁ *+a _(A2) QRA ₂ *+ . . . +a _(An) QRA _(n)*+a _(B1) QRB ₁ +a _(B2) QRB ₂ *+ . . . +a _(Bn) QRB _(n)  (9)where QAx₀* is one of four dimensionless end airflows for a cluster andthe coefficients a_(Ai) and a_(Bi) effectively weight the relativeimportance of each rack on the end airflow. The weighting coefficientsassociated with racks located near the end of the row will be muchlarger than those associated with more interior racks. Further,empirically it is found that only racks in the first four or fivepositions nearest the end of the row need be retained in Equation 9. Forthe fixed conditions considered, the constant ao is negative, implyingthat the flow is “out” (driven by buoyancy) when there is zero rackairflow.

To determine the values of the coefficients in Equation 9 for aparticular set of room environment and cluster geometry, many (on theorder of 100) CFD simulations may be performed at a number of differentperforated tile flow rates. A large pool of rack power values may becreated from which the many CFD simulations draw rack power and airflowdata from, either randomly or systematically. The rack power values maybe based on the frequency distribution of actual data center racks asdetermined from a survey. The rack power and airflow values used in theCFD simulations may be scaled as necessary to achieve practical totalsupply-to-equipment airflow ratios in the range of, for example, 0.9 to3 for each perforated tile flow rate considered. The CFD data is thenused to determine a least-squares fit of the coefficients in Equation 9for each tile flow rate considered.

In summary, a simple end airflow model has been described whichaccurately accounts for a non-uniform distribution of rack airflow andpower for a fixed set of room conditions. In at least one embodiment,the model is generalized to include the effects of geometricenvironment, the thermal environment, and supply airflow rate. Theeffects of the end airflow penetrate only a few rack distances down therow; for longer row lengths predictions for the majority of the racks inthe cluster will be good even if the end airflow model is not asaccurate as desired.

The airflow into or out of the top of each control volume has been left“floating” as necessary degrees of freedom in the above example. Now,with all of the horizonital airflows computed as discussed above, theairflow at the top of each control volume is computed based on theconservation of mass. With reference to FIG. 13, using dimensionalquantities, the equations for A-row and B-row control volumes aredetermined using equations 9(a) and 9(b).QAtop_(i) =Q _(T) −QRA _(i) +QAx _(i−1) −Qz _(i) −QAx _(i)  (10a)QBtop_(i) =Q _(T) −QRB _(i) +QBx _(i−1) +Qz _(i) −QBx _(i)  (10b)Applied to all control volumes, equations 9a and 9b represent a total of2n equations. At this stage, there is only one unknown per equation(QAtop_(i) and QBtop_(i)) so they may be solved sequentially.

At this point, all airflows within the cold aisle are known for theexample. What remains is to track the airflow into each rack so that itsorigin may be identified and the recirculation index (RI) can becalculated for each rack. As discussed above, RI is the fraction ofrecirculated air ingested by a rack. The recirculated air can enter thecold aisle at any point where there is inflow at the ends of the rows oralong the top of the cold aisle. Further, the warm recirculated air neednot directly enter the cold aisle via the control volume immediatelyadjacent to a rack of interest; it may enter anywhere, travel anywherethe airflow patterns take it, and end up at the inlet of any rack.

To compute RI for each rack the cool supply air is distinguished fromthe warm recirculated air at all points in the cold aisle.Mathematically, this is accomplished by defining the concentration ofrecirculated air at any point in the cold aisle using Equation 11.C _(recirc)=(mass of recirculated air)/(total mass of air)  (11)It follows from Equation 11 that the supply airflow emerging from thetiles has a C_(recirc)=0 and that anywhere the recirculated air entersthe cold aisle along the sides or top of the cold aisle and C_(recirc)may be set equal to 1. In practice, C_(recirc) may be set to a valueless than 1 for the ends of the cold aisle recognizing that, on average,the top is generally much warmer than the ends of the cold aisle.Accordingly, in one embodiment, C_(recirc)=0.5 for any inflow at theends of the cold aisle.

The recirculated air can be assumed to have the same physical propertiesas the cool supply air so that it has no effect, e.g. due to a densitydifference, on the airflow patterns in the cold aisle.

Now consider a small volume just covering a rack inlet. Equation 11applied to this volume represents the average C_(recirc) over thisvolume. Dividing the numerator and denominator by a small time incrementAt and taking the limit as Δt→0, demonstrates that the averageC_(recirc) over a rack inlet is precisely the rack recirculation index.Thus, to determine the RR's for each rack the average C_(recirc) overeach rack inlet is determined. Referring back to FIG. 8, we can estimatethe RR for each rack as the average C_(recirc) of the control volumeimmediately adjacent to the rack of interest. C_(recirc) over all 2ncontrol volumes can be computed from the conservation of mass of therecirculated air using Equation 12. $\begin{matrix}{{\sum\limits_{\underset{Faces}{{All}\quad{CV}}}^{\quad}{C_{recirc}Q}} = 0} & (12)\end{matrix}$where Q is the total airflow rate through each control volume face andis a known value at this stage of the calculation.

FIG. 13, shows control volumes 1008 and 1010 of a transverse section ofa cold aisle 1006. Equation 12 is applied to the control volumes 1008and 1010. For convenience, we label the C_(recirc) crossing each controlvolume surface with same convention used for airflows while dropping the“recirc” subscript. The result isC _(T) Q _(T)+(CAx _(i−1))(QAx _(i−1))=(CRA _(i))(QRA _(i))+(CAx_(i))(QAx _(i)) +(Cz _(i))(Qz _(i))+(CAtop_(i))(QAtop_(i))  (13a)C _(T) Q _(T)+(CBx _(i−1))(QBx _(i−1))+(Cz _(i))(Qz _(i))+(CRB _(i))(QRB_(i)) +(CBxhd i)(QBx _(i))+(CBtop_(i))(QBtop_(i))  (13b)

Equations 13a and 13b are not solved directly because the number OfC_(recirc) values exceeds the number of equations. Estimating eachC_(recirc) as the average C_(recirc) from the “upwind” control volume,results in a proper balance of 2n unknown C_(recirc)'s and 2n equations.Based on this “upwind” approach, the appropriate C_(recirc) values areinserted into Equations 13a and 13b after the airflow patterns in thecold aisle have been computed thereby establishing the direction ofairflow crossing each face of each control volume. TABLE 4 C_(recirc)Settings Based on Airflow Direction Upwind Value of C_(recirc) AirflowAirflow ≧ 0 Airflow < 0 Q_(t) 0 0 QAx_(i) CA_(i) CA_(i+1) QBx_(i) CB_(i)CB_(i+1) Qz_(i) CA_(i) CB_(i) QAtop_(i) CA_(i) 1 QBtop_(i) CB_(i) 1

Table 4 shows the appropriate upwind values of C_(recirc) to be used inEquations 13a and 13b where the CA_(i) and CB_(i) are the averageC_(recirc) over the relevant “A” or “B” control volumes respectively.Not shown in the table are the settings for QAx_(i) and QBx_(i) at theend of the row, i.e. Qax₀, QBx₀, QAx_(n), and QBx_(n). In this caseC_(recirc) may be set to 0.5 as discussed above for any “inflow”.

With the values of C_(recirc) taken from Table 4, the 2n Equationsrepresented by 13a and 13b may be solved simultaneously for the 2nCA_(i) and CB_(i) values. These simple linear equations can be solvedwithout iteration virtually instantaneously for any practical row lengthusing common computing hardware. Finally, as discussed above, thecomputed CA_(i) and CB_(i) values may be directly interpreted as therecirculation index of the adjacent “A” and “B” racks respectively.

In other embodiments, because of the similarity between the energy andconcentration equations, bulk average temperature could be determinedover each control volume instead of RI following a very similarprocedure.

A summary of a process 1050 for determining recirculation index for acluster of racks using the methodology described above will now beprovided with reference to FIG. 14. In a first stage 1052 of theprocess, the row length, tile airflow, rack airflow and rack power aredefined for a cold aisle to be analyzed. Next, in stage 1054, empiricaldata used for computing airflows is imported from a CFD analysis asdescribed above. The end airflows are then determined at stage 1056based on details of the cluster and details of the room environment. Allhorizontal airflows are then determined at stage 1058. At stage 1060,horizontal airflows induced by the 4 end airflows are computed, and atstage 1062, complete horizontal airflows are computed by adding theairflows from stages 1058 and 1060. Vertical airflows are computed atstage 1064, and then at stage 1066, the recirculation index may bedetermined for each rack by solving a set of conservation equations forthe recirculated air as described above.

In one embodiment, to determine cooling capacity for a given rack basedon the recirculation index, a threshold recirculation index is firstestablished, below which a design is considered unsatisfactory. For eachrack, after a satisfactory design is achieved, the power of the rack isincreased until the recirculation index of the that rack (or any otherrack) reaches the threshold level, and the power at which that occursrepresents the maximum cooling capacity for the rack. A similar methodfor determining cooling capacity can be used with other analysesdescribed herein, including the analysis using capture index valuesdescribed below.

In other embodiments, the control volume and superposition methodsdescribed above may be modified. These modifications may include the useof more complex statistical methods (e.g., the use of neural networks)to determine end airflow conditions from large pools of CFD data.Further, the number of control volumes may be substantially increased toimprove accuracy and resolution of computed variables. In particular,the latter improvement would allow airflow variations at various rackelevations (e.g., due to a variety of equipment installed in a rack) tobe considered. The basic methodology could further be modified toinclude layouts beyond the scope discussed above including layoutsinvolving an arbitrary number of perforated tiles of arbitrary flowrate, an arbitrary cold aisle width, arbitrary rack dimensions or othersuch variations from examples discussed above.

In processes described above, cooling analyses of a data center havefocused primarily on determining airflows in the cool aisle for acluster of racks located in a data center having a raised floor.Embodiments described above, however, are not limited for use in datacenters having raised floors, and aspects of the embodiments are alsoapplicable to data centers that do not include raised floor coolingsystems. At least one embodiment described above provides a decouplingof the cold aisle from the remainder of the data center to computeairflows in the cold aisle. The effect of the room environment is thenbuilt back into the analysis using end-of-row airflows that arecomputed, for example, in separate CFD calculations that may be computedoffline and made available through look-up tables or empiricalcorrelations. As described below, in a similar manner to that describedabove, a hot aisle in a data center can be analyzed by decoupling theaisle from the remainder of the room and later building the effects ofthe room environment back into the analysis.

In additional embodiments that will now be described, processes areprovided for evaluating a cluster of racks based on airflows that occurin a hot aisle for a cluster of racks. In at least one version of theadditional embodiments, a raised floor data center is not used, butrather, cooling is provided using in-row cooling units as describedabove. In one particular process of one embodiment, a capture index (CI)is calculated and used to analyze a cluster of racks in a data center.The capture index is used in one embodiment with a row or cluster ofracks having one or more in-row cooling units, and the capture index isdefined as the percentage of air released by a rack into a hot aisle,which is captured by cooling units bounding the hot aisle. The CI may beconsidered as a complementary metric to the RI described above for usewith the hot aisle. The CI is useful when the focus of a design is tokeep the hot air within the hot aisle. As discussed above, rack inlettemperatures are typically the ultimate cooling metric, however, if allof the hot air is captured in the hot aisle, the rest of the data center(including rack inlets) can be designed and controlled to remain at“room temperature.”

In one embodiment, the use of chemical concentrations with, for examplea CFD analysis, can be used to quantitatively determine CI. The exhaustof each rack is identified in such an analysis as a separate specieshaving the same properties as air, so as not to change the physics ofairflow. The fraction of hot air released from racki (identified asC^(i)) which is captured by an in-row cooler identified as coolerj maybe computed using Equation 14 below.f _(ij) =C _(j) ^(i)(Q _(coolerj))/Q _(racki))  (14)

where:

-   -   C_(ji) is the concentration of C^(i) at the inlet of cooler j    -   Q_(coolerj) is the airflow rate (e.g. in cfm) through coolerj    -   Q_(racki) if the airflow rate (e.g. in cfm) through racki

As an example, if the cooler and rack airflow are equal, and theconcentration of exhaust air C^(i) from rack i at the cooler inlet ismeasured to be 0.5, then this implies that half of the exhaust air fromrack i is captured by cooler j. In a hot aisle having N coolers, thenthe capture index (CI) is the sum of all of the f_(ij)'s over all Ncoolers and can be expressed using Equation 15 below. $\begin{matrix}{{CI}_{i} = {\sum\limits_{j = 1}^{N}{C_{j}^{i}\frac{Q_{{cooler}\quad j}}{Q_{{rack}\quad i}}}}} & (15)\end{matrix}$

As will now be described, with reference to FIG. 15, which shows acluster of racks 1080, a set of empirical rules can be used to determinethe CI for each rack 1082 of the cluster. As shown in FIG. 15, theresulting CI values may be displayed on a display with the associatedracks. In one example, racks having a CI less than 60% are identified inred indicating a warning, racks having a CI between 60% and 80% areindicated in yellow as a caution, and racks having a CI greater than 80%are indicated in green indicating that the CI is satisfactory.

In one embodiment, a large pool of CFD runs can be performed toestablish and refine empirical rules. In other embodiments, neuralnetworks and other techniques may be used to refine rules. The cluster1080 includes two rows (row A and row B) of parallel racks, that exhaustair to a common hot aisle 1084. Each Rack is labeled A1-A6 and B1-B7,identifying the row and position in the row of the rack, and for theexample shown each rack has a power draw of 2 kW. The cluster alsoincludes in-row cooling units 1086. In FIG. 15, a number of half-rackcooling units 1086 are shown, but embodiments of the invention may alsobe used with full-width rack cooling units or other devices. Thehalf-rack cooling units used in the example associated with FIG. 15 havea nominal cooling capacity of 17 kW. Also shown in FIG. 15 is the CI interms of percentage for each rack. The CI is shown as a percentage andindicates for each rack, the percentage of its exhaust air that iscaptured by one of the cooling units.

The CI is determined based on the concept that all rack-cooling unitinteractions depend only on the magnitude of airflow associated with theracks and cooling units and their relative geometric positions. Eachrack location can be thought of as having a certain potential to supplyairflow to other rack locations. This potential varies inversely withthe distance of separation. For example, rack A1 in FIG. 15 couldpotentially supply a large portion of its airflow to the area near rackA2. However, much less of rack A1's airflow could make its way toposition A6. Further, the amount of airflow a rack can supply to otherlocations is in direct proportion to its own total airflow. The netairflow which can be supplied to a particular rack location A1 can berepresented using Equation 16 below. $\begin{matrix}{( Q_{Ai} )_{net} = {( Q_{Ai} )_{self} + {\sum\limits_{{all}\quad{other}\quad{racks}\quad j}^{\quad}{( Q_{Aj} )_{self}A\quad{\mathbb{e}}^{{- B}\quad\Delta\quad x}}} + {c\{ {( Q_{Bi} )_{self} + {\sum\limits_{{all}\quad{other}\quad{racks}\quad j}^{\quad}{( Q_{Bj} )_{self}A\quad{\mathbb{e}}^{{- B}\quad\Delta\quad x}}}} \}}}} & (16)\end{matrix}$

where

-   -   (Q_(Ai))_(net)=The net maximum airflow that can be supplied to        location Ai including contributions from all other racks.    -   (Q_(Ai))_(self)=The actual airflow supplied by the rack at        location Ai.    -   A=empirical constant.    -   B=empirical constant.    -   C=empirical “coupling” constant for accounting for effects from        opposite row.

The net maximum airflow that can be supplied to various locations in theB row is computed using a similar expression. Finally, the sameexpression is used to compute the net maximum airflow, which can becaptured at any rack location—with the sum over all cooler instead ofracks. The CI is then estimated as the ratio of net airflow captured andnet airflow supplied at any location expressed as a percentage and withvalues capped at 100%. The constants A, B, and C are selected to providethe best statistical fit to benchmark CFD data. Different values of theconstants may be used to account for alternative configurationsincluding different cooler types, different average rack power orpeak-to-average power ratios and alternative hot-aisle spacing, rowlengths, or room environments. As an example, consider a cluster ofaverage power racks with modest rack-to-rack power variations. Thecluster is 14 ft. long, contains a 3 ft. wide hot aisle, and is assumedto be in a fairly tightly packed data center environment with a 12 ft.ceiling height. In this case, reasonable predictions are made with theempirical constants taken as A=0.56, B=0.33, and C=0.65.

In the embodiment described above, the CI is calculated for a cluster ofracks having uniform depth and width. In other embodiments, the sameprocesses may be used for racks of non-uniform depth and width. In oneembodiment, the CI calculations described above are programmed into aMicrosoft Excel Spreadsheet program that allows a user to add and movecooling units to see the effect of different numbers of cooling unitsand their placements. In other embodiments, the process for determiningCI described above may be incorporated into data center design andmanagement systems, such as system 200 discussed above.

In the embodiment above, an exponential expression is used for modelingthe rack and cooler interactions. In other embodiments, otherexpressions may be used, such as a polynomial or any other mathematicalexpression which contains a number of parameters which may be tuned toprovide the best fit to benchmark performance data. Further, differentcurves and/or different coefficients may be used for the portion of thecalculation associated with determining air supplied by racks than usedin the portion of the calculation used for determining air captured bycooling units. In another embodiment, the rules may be further refinedto address specific situations. For example, a Rack A may have no effecton another Rack B where a third rack C is located between Rack A andRack B and has greater airflow than either Rack A or Rack B.

In still another embodiment, the effects of the ends of rows may beaccounted for explicitly. Separate CFD simulations may be conducted todetermine the net inflow or outflow of air at each end of a row forgiven layouts of racks and cooling units. The results of the CFDsimulations may be incorporated into the empirical methods describedabove to determine CI for racks in a cluster. The results of the CFDsimulations can be used to provide correct airflow estimates at the endsof the row, while one of the algorithms discussed above may be used todetermine CI at more interior portions of the row. Similarly, theeffects of a missing rack or racks may be simulated using CFD with theresults incorporated into the empirical methods.

The capture index method of analysis discussed above provides arack-by-rack, or local, cooling performance metric for equipment racksin a data center. In addition to using CI as a rack-level metric in ahot aisle analysis, in another embodiment, cluster-wide performancemetrics are determined, and the overall cooling performance of thecluster may be determined based on both the CI metric and the globalmetric. The cluster-wide performance metrics indicate whether thecluster as a whole will have adequate cooling performance. The CIidentifies which racks are not having their exhaust airflow adequatelycaptured. If a rack having a low CI is a low power rack, this may notresult in a problem. Furthermore, a rack may have a relatively high CI,yet still cause problems if it is a high power rack with a high outlettemperature. In one embodiment, the global metric that is used is adetermination of the net power which escapes the hot aisle. The netescaped power can be determined using equation 17. $\begin{matrix}{\sum\limits_{{all}\quad{racks}\quad i}^{\quad}{( {1 - {CI}_{i}} )P_{i}}} & (17)\end{matrix}$

-   -   where,    -   CI_(i)=the capture index for rack i expressed as a fraction        (rather than percentage), and    -   P_(i)=the power of rack i.

The net escaped power determined by equation 17 may be correlated tomaximum cluster inlet temperatures (e.g., a net escaped power of 25 kWmay imply a maximum cluster rack inlet temperature of seventy-ninedegrees F) for clusters of particular geometric layouts (e.g., hot aislewidth, row length, etc.), room environments and rack and cooler details(e.g., cooling unit flow rate and rack airflow/power (cfm/kW)).Accordingly, the net escaped power may be used to determine the highestrack inlet temperature.

In other embodiments, other techniques described above for calculatingairflows in a cool aisle may be applied to a hot aisle analysis todetermine CI, including the use of CFD, CGCFD and control volumes. Theuse of superposition may be less applicable in hot aisle analysesbecause the airflow patterns are not ideal. In still another embodiment,a process 1100 is provided for determining CI in a hot aisle using a CFDanalysis. The CFD analysis is performed only on the hot aisle itself andend of row airflows are determined separately and may be patched intothe CFD solution. Common computer hardware can be used to conduct such aCFD analysis in 10 to 20 seconds. The process 1100 is shown in flowchartform in FIG. 16. In a first stage 1102 of the process 1100 data relatedto the layout is loaded either manually, read from a database, or in anyother manner. The data related to the layout may include row length,power draw and airflow for each rack, rack dimensions, cooling unittypes, locations and flow rates, hot aisle widths and room environmentdetails (i.e., size, temperatures).

At stage 1104 of the process 1100, hot-aisle end of row airflows aredetermined using best-fit curves, or look-up tables based on prior CFDstudies. Hot aisle end of row calculations may be performed using thesame techniques used to determine cold aisle end of row airflows withthe input rack airflow provided as a positive value and the coolingunits airflow input as a negative value. At stage 1106, a CFD analysisofjust the hot aisle is performed using results of the end airflowanalysis of stage 1104 and with the top of the hot aisle taken as aconstant-pressure boundary. In the analysis, the exhaust air for eachrack is “tagged” with a particular concentration. At stage 1108, the CIfor each rack is determined based on the ratio of captured and suppliedairflows computed by equation 16 and the results of stage 1106. Theglobal cluster cooling metric may also be determined at this point usingequation 17. At stage 1110, the rack CI's and the global cluster coolingmetric can be used to determine if cooling for the cluster is adequate.

In addition to the cooling analysis methods discussed above, whichprovide real-time analysis of data centers, additional cooling analysismethods may be used in embodiments of the invention. These additionalmethods include a panel method, a potential analysis method, flownetwork/zonal modeling, principal component analysis or a combination ofany of these and the methods discussed above. These additional coolinganalysis methods are generally known to those of skill in the art.

Various embodiments according to the invention may be implemented on oneor more computer systems as discussed above. For example, system 200 maybe implemented in a single computer system or in multiple computersystems. These computer systems may be, for example, general-purposecomputers such as those based on Intel PENTIUM-type processor, MotorolaPowerPC, Sun UltraSPARC, Hewlett-Packard PA-RISC processors, or anyother type of processor.

For example, various aspects of the invention may be implemented asspecialized software executing in a general-purpose computer system 900such as that shown in FIG. 17. The computer system 900 may include aprocessor 903 connected to one or more memory devices 904, such as adisk drive, memory, or other device for storing data. Memory 904 istypically used for storing programs and data during operation of thecomputer system 900. The computer system 900 may also include a storagesystem 906 that provides additional storage capacity. Components ofcomputer system 900 may be coupled by an interconnection mechanism 905,which may include one or more busses (e.g., between components that areintegrated within a same machine) and/or a network (e.g., betweencomponents that reside on separate discrete machines). Theinterconnection mechanism 905 enables communications (e.g., data,instructions) to be exchanged between system components of system 900.

Computer system 900 also includes one or more input devices 902, forexample, a keyboard, mouse, trackball, microphone, touch screen, and oneor more output devices 907, for example, a printing device, displayscreen, speaker. In addition, computer system 900 may contain one ormore interfaces (not shown) that connect computer system 900 to acommunication network (in addition or as an alternative to theinterconnection mechanism 905).

The storage system 906, shown in greater detail in FIG. 18, typicallyincludes a computer readable and writeable nonvolatile recording medium911 in which signals are stored that define a program to be executed bythe processor or information stored on or in the medium 911 to beprocessed by the program to perform one or more functions associatedwith embodiments described herein. The medium may, for example, be adisk or flash memory. Typically, in operation, the processor causes datato be read from the nonvolatile recording medium 911 into another memory912 that allows for faster access to the information by the processorthan does the medium 911. This memory 912 is typically a volatile,random access memory such as a dynamic random access memory (DRAM) orstatic memory (SRAM). It may be located in storage system 906, as shown,or in memory system 904. The processor 903 generally manipulates thedata within the integrated circuit memory 904, 912 and then copies thedata to the medium 911 after processing is completed. A variety ofmechanisms are known for managing data movement between the medium 911and the integrated circuit memory element 904, 912, and the invention isnot limited thereto. The invention is not limited to a particular memorysystem 904 or storage system 906.

The computer system may include specially-programmed, special-purposehardware, for example, an application-specific integrated circuit(ASIC). Aspects of the invention may be implemented in software,hardware or firmware, or any combination thereof. Further, such methods,acts, systems, system elements and components thereof may be implementedas part of the computer system described above or as an independentcomponent.

Although computer system 900 is shown by way of example as one type ofcomputer system upon which various aspects of the invention may bepracticed, it should be appreciated that aspects of the invention arenot limited to being implemented on the computer system as shown in FIG.17. Various aspects of the invention may be practiced on one or morecomputers having a different architecture or components shown in FIG.17. Further, where functions or processes of embodiments of theinvention are described herein (or in the claims) as being performed ona processor or controller, such description is intended to includesystems that use more than one processor or controller to perform thefunctions.

Computer system 900 may be a general-purpose computer system that isprogrammable using a high-level computer programming language. Computersystem 900 may be also implemented using specially programmed, specialpurpose hardware. In computer system 900, processor 903 is typically acommercially available processor such as the well-known Pentium classprocessor available from the Intel Corporation. Many other processorsare available. Such a processor usually executes an operating systemwhich may be, for example, the Windows 95, Windows 98, Windows NT,Windows 2000 (Windows ME) or Windows XP operating systems available fromthe Microsoft Corporation, MAC OS System X operating system availablefrom Apple Computer, the Solaris operating system available from SunMicrosystems, or UNIX operating systems available from various sources.Many other operating systems may be used.

The processor and operating system together define a computer platformfor which application programs in high-level programming languages arewritten. It should be understood that embodiments of the invention arenot limited to a particular computer system platform, processor,operating system, or network. Also, it should be apparent to thoseskilled in the art that the present invention is not limited to aspecific programming language or computer system. Further, it should beappreciated that other appropriate programming languages and otherappropriate computer systems could also be used.

One or more portions of the computer system may be distributed acrossone or more computer systems coupled to a communications network. Forexample, as discussed above, a computer system that performs build-outfunctions may be located remotely from a system manager. These computersystems also may be general-purpose computer systems. For example,various aspects of the invention may be distributed among one or morecomputer systems configured to provide a service (e.g., servers) to oneor more client computers, or to perform an overall task as part of adistributed system. For example, various aspects of the invention may beperformed on a client-server or multi-tier system that includescomponents distributed among one or more server systems that performvarious functions according to various embodiments of the invention.These components may be executable, intermediate (e.g., IL) orinterpreted (e.g., Java) code which communicate over a communicationnetwork (e.g., the Internet) using a communication protocol (e.g.,TCP/IP). For example, one or more database servers may be used to storedevice data that is used in designing layouts, and one or more serversmay be used to efficiently perform cooling calculations associated withembodiments of the present invention.

It should be appreciated that the invention is not limited to executingon any particular system or group of systems. Also, it should beappreciated that the invention is not limited to any particulardistributed architecture, network, or communication protocol.

Various embodiments of the present invention may be programmed using anobject-oriented programming language, such as SmallTalk, Java, C++, Ada,or C# (C-Sharp). Other object-oriented programming languages may also beused. Alternatively, functional, scripting, and/or logical programminglanguages may be used. Various aspects of the invention may beimplemented in a non-programmed environment (e.g., documents created inHTML, XML or other format that, when viewed in a window of a browserprogram, render aspects of a graphical-user interface (GUI) or performother functions). Various aspects of the invention may be implemented asprogrammed or non-programmed elements, or any combination thereof.

In embodiments of the invention discussed above, systems and methods aredescribed that provide indications of remaining cooling capacity forequipment enclosures. The indication of remaining cooling capacity maybe a direct indication of remaining cooling in terms of, for example,kilowatts or BTU per hour, or the indication may be indirect such asproviding the total capacity of cooling available to an enclosure alongwith an indication of how much cooling is being used, for example, interms of percentage. Further, calculated values, including the captureindex and the recirculation index may be used to determine thesufficiency of a particular design and to determine additional coolingcapacity before a warning or error condition will result.

Embodiments of a systems and methods described above are generallydescribed for use in relatively large data centers having numerousequipment racks, however, embodiments of the invention may also be usedwith smaller data centers and with facilities other than data centers.Further, as discussed above, embodiments of the present invention may beused with facilities having raised floors as well as with facilitiesthat do not have a raised floor.

In embodiments of the present invention discussed above, results ofanalyses are described as being provided in real-time. As understood bythose skilled in the art, the use of the term real-time is not meant tosuggest that the results are available immediately, but rather, areavailable quickly giving a designer the ability to try a number ofdifferent designs over a short period of time, such as a matter ofminutes.

Having thus described several aspects of at least one embodiment of thisinvention, it is to be appreciated various alterations, modifications,and improvements will readily occur to those skilled in the art. Suchalterations, modifications, and improvements are intended to be part ofthis disclosure, and are intended to be within the spirit and scope ofthe invention. Accordingly, the foregoing description and drawings areby way of example only.

1. A computer-implemented method for evaluating the cooling performanceof a cluster of equipment racks in a data center, wherein the cluster ofequipment racks includes at least a first row of racks and a second rowof racks separated by a cool aisle, with each of the equipment racksbeing configured to draw cooling air from the cool aisle, the methodcomprising: obtaining at least one of power data and airflow data foreach of the equipment racks; obtaining cool airflow data for cool airsupplied to the cool aisle from a source of cool air; and conducting ananalysis of airflows in the cool aisle to determine a recirculationindex for at least one of the equipment racks, wherein the recirculationindex is indicative of a quantity of recirculated air included in aninput airflow of the at least one equipment rack.
 2. The method of claim1, wherein the recirculation index is equal to a ratio of recirculatedair to total air in the input airflow of the at least one equipmentrack.
 3. The method of claim 2, further comprising determining arecirculation index for each of the equipment racks.
 4. The method ofclaim 2, wherein the act of obtaining cool airflow data includesobtaining cool airflow data for an in-row cooling unit included in thecluster of racks.
 5. The method of claim 2, wherein the act of obtainingcool airflow data includes obtaining cool airflow data of at least oneperforated tile included in the cool aisle.
 6. The method of claim 1,wherein the act of conducting an analysis includes defining a pluralityof control volumes in the cool aisle, and wherein the method furtherincludes determining airflows in the cool aisle by determining airflowinto and out of at least one of the control volumes.
 7. The method ofclaim 3, further comprising comparing the recirculation index for eachof the plurality of equipment enclosures with a threshold.
 8. The methodof claim 7, further comprising determining a cooling capacity for eachof the equipment enclosures based on the recirculation index for each ofthe equipment enclosures.
 9. The method of claim 8, further comprisingdisplaying the cooling capacity for each of the equipment enclosuresalong with a representation of a data center containing the cluster. 10.The method of claim 1, wherein the act of conducting an analysisincludes assigning different chemical concentration identifiers to theairflows for at least two of the plurality of equipment racks.
 11. Themethod of claim 1, wherein the act of conducting an analysis includesimporting empirical data and determining end of aisle airflows using theempirical data.
 12. The method of claim 11, wherein the act ofconducting an analysis includes determining cool aisle airflows inisolation from the data center to obtain isolated results, and combiningthe isolated results with the empirical data.
 13. A computer-implementedmethod for evaluating the cooling performance of a cluster of equipmentracks in a data center, wherein the cluster of equipment racks includesat least a first row of racks and a second row of racks separated by ahot aisle, with each of the equipment racks being configured to exhaustair into the hot aisle, the method comprising: obtaining at least one ofpower data and airflow data for each of the equipment racks; obtainingairflow data for at least one air removal unit contained in one of thefirst row of equipment racks and the second row of equipment racks; andconducting an analysis of airflows in the hot aisle to determine acapture index for at least one of the equipment racks, wherein thecapture index is indicative of a fraction of air that is exhausted bythe at least one of the equipment racks and captured by the at least oneair removal unit.
 14. The method of claim 13, wherein the at least oneair removal unit includes an in-row cooling unit.
 15. The method ofclaim 14, wherein the capture index is equal to a ratio of captured airto total air exhausted by the at least one equipment rack.
 16. Themethod of claim 15, further comprising determining a capture index foreach of the equipment racks.
 17. The method of claim 16, wherein the actof conducting an analysis includes defining a plurality of controlvolumes in the hot aisle, and wherein the method further includesdetermining airflows in the hot aisle by determining airflow into andout of at least one of the control volumes.
 18. The method of claim 16,further comprising comparing the capture index for each of the pluralityof equipment enclosures with a threshold.
 19. The method of claim 18,further comprising determining a cooling capacity for each of theequipment enclosures based on the capture index for each of theequipment enclosures.
 20. The method of claim 19, further comprisingdisplaying the cooling capacity for each of the equipment enclosuresalong with a representation of a data center containing the cluster. 21.The method of claim 13, wherein the act of conducting an analysisincludes assigning different chemical concentration identifiers to theairflows for at least two of the plurality of equipment racks.
 22. Themethod of claim 13, wherein the act of conducting an analysis includesimporting empirical data and determining end of aisle airflows using theempirical data.
 23. The method of claim 22, wherein the act ofconducting an analysis includes determining hot aisle airflows inisolation from the data center to obtain isolated results, and combiningthe isolated results with the empirical data.
 24. The method of claim13, wherein the act of conducting an analysis includes importingempirical rules, and determining the capture index using the empiricalrules.
 25. The method of claim 24, wherein the empirical rules includecoefficients for use in determining at least one capture index.
 26. Acomputer-readable medium encoded with instructions for execution on acomputer system, the instructions when executed, performing a methodcomprising acts of: obtaining at least one of power data and airflowdata for a plurality of equipment racks arranged in a cluster, whereinthe cluster of equipment racks includes at least a first row of racksand a second row of racks separated by a cool aisle, with each of theequipment racks being configured to draw cooling air from the coolaisle; obtaining cool airflow data for cool air supplied to the coolaisle from a source of cool air; and conducting an analysis of airflowsin the cool aisle to determine a recirculation index for at least one ofthe equipment racks, wherein the recirculation index is indicative of aquantity of recirculated air included in an input airflow of the atleast one equipment rack.
 27. The computer-readable medium of claim 26,wherein the recirculation index is equal to a ratio of recirculated airto total air in the input airflow of the at least one equipment rack.28. The computer-readable medium of claim 27, wherein the acts furtherinclude determining a recirculation index for each of the equipmentracks.
 29. The computer-readable medium of claim 27, wherein the act ofobtaining cool airflow data includes obtaining cool airflow data for anin-row cooling unit included in the cluster of racks.
 30. Thecomputer-readable medium of claim 27, wherein the act of obtaining coolairflow data includes obtaining cool airflow data of at least oneperforated tile included in the cool aisle.
 31. The computer-readablemedium of claim 26, wherein the act of conducting an analysis includesdefining a plurality of control volumes in the cool aisle, and whereinthe method further includes determining airflows in the cool aisle bydetermining airflow into and out of at least one of the control volumes.32. The computer-readable medium of claim 28, further comprisingcomparing the recirculation index for each of the plurality of equipmentenclosures with a threshold.
 33. The computer-readable medium of claim32, wherein the acts further include determining a cooling capacity foreach of the equipment enclosures based on the recirculation index foreach of the equipment enclosures.
 34. The computer-readable medium ofclaim 33, wherein the acts further include displaying the coolingcapacity for each of the equipment enclosures along with arepresentation of a data center containing the cluster.
 35. Thecomputer-readable medium of claim 26, wherein the act of conducting ananalysis includes assigning different chemical concentration identifiersto the airflows for at least two of the plurality of equipment racks.36. The computer-readable medium of claim 26, wherein the act ofconducting an analysis includes importing empirical data and determiningend of aisle airflows using the empirical data.
 37. Thecomputer-readable medium of claim 36, wherein the act of conducting ananalysis includes determining cool aisle airflows in isolation from thedata center to obtain isolated results, and combining the isolatedresults with the empirical data.
 38. A computer-readable medium encodedwith instructions for execution on a computer system, the instructionswhen executed, performing a method comprising acts of: obtaining atleast one of power data and airflow data for a plurality of equipmentracks arranged in a cluster, wherein the cluster of equipment racksincludes at least a first row of racks and a second row of racksseparated by a hot aisle, with each of the equipment racks beingconfigured to exhaust air into the hot aisle; obtaining airflow data forat least one air removal unit contained in one of the first row ofequipment racks and the second row of equipment racks; and conducting ananalysis of airflows in the hot aisle to determine a capture index forat least one of the equipment racks, wherein the capture index isindicative of a fraction of air that is exhausted by the at least one ofthe equipment racks and captured by the at least one air removal unit.39. The computer-readable medium of claim 38, wherein the at least oneair removal unit is an in-row cooling unit.
 40. The computer-readablemedium of claim 39, wherein the capture index is equal to a ratio ofcaptured air to total air exhausted by the at least one equipment rack.41. The computer-readable medium of claim 40, wherein the acts furtherinclude determining a capture index for each of the equipment racks. 42.The computer-readable medium of claim 41, wherein the act of conductingan analysis includes defining a plurality of control volumes, andwherein the method further includes determining airflows in the hotaisle by determining airflow into and out of at least one of the controlvolumes.
 43. The computer-readable medium of claim 41, wherein the actsfurther include comparing the capture index for each of the plurality ofequipment enclosures with a threshold.
 44. The computer-readable mediumof claim 43, wherein the acts further include determining a coolingcapacity for each of the equipment enclosures based on the capture indexfor each of the equipment enclosures.
 45. The computer-readable mediumof claim 44, wherein the acts further include displaying the coolingcapacity for each of the equipment enclosures along with arepresentation of a data center containing the cluster.
 46. Thecomputer-readable medium of claim 39, wherein the act of conducting ananalysis includes assigning different chemical concentration identifiersto the airflows for at least two of the plurality of equipment racks.47. The computer-readable medium of claim 39, wherein the act ofconducting an analysis includes importing empirical data and determiningend of aisle airflows using the empirical data.
 48. Thecomputer-readable medium of claim 47, wherein the act of conducting ananalysis includes determining hot aisle airflows in isolation from thedata center to obtain isolated results, and combining the isolatedresults with the empirical data.
 49. The computer-readable medium ofclaim 39, wherein the act of conducting an analysis includes importingempirical rules, and determining the capture index using the empiricalrules.
 50. The computer-readable medium of claim 49, wherein theempirical rules include coefficients for use in determining at least onecapture index.